Introduction to Binary Exploitation

Binary exploitation (“pwn”) consists of leveraging bugs in compiled programs —typically C/C++— to alter their execution and, ultimately, run your own code or take control of the process. Unlike the web, here you work at the level of memory, registers, and assembly: a mistake handling a buffer, a pointer, or an integer becomes control of the execution flow. It’s the flagship discipline of CTFs and the basis of low-level software exploits.

Why these bugs exist

C and C++ give full control over memory but with no safety net: they don’t check array bounds, trusting the programmer to manage pointers and sizes correctly. A mistake (writing past a buffer, using freed memory, trusting an attacker-controlled size) corrupts memory structures the attacker can manipulate to redirect execution.

Base concepts

# a process's memory (simplified)
Stack     local variables, return addresses (grows downward)
Heap      dynamic memory (malloc/free)
.text     program code (executable)
.data/.bss global variables
# key registers (x86-64)
RIP/PC    instruction pointer: which instruction runs (the goal)
RSP       stack pointer   RBP   base pointer
RAX..R15  general-purpose registers

The goal is almost always to control RIP (the instruction pointer): if you decide what runs next, you control the program.

The vulnerability classes

Stack buffer overflow   write past a stack buffer -> overwrite the return address (see pwn-stack)
Format string           printf(user) -> arbitrary memory read/write (pwn-format)
Heap exploitation       corrupt heap metadata (use-after-free, overflow) (pwn-heap)
Integer issues          overflow/underflow/signedness -> wrong sizes/indices (pwn-integer)
Use-after-free          use already-freed memory

Modern protections (mitigations)

Current binaries have defenses you must understand and bypass:

ASLR          randomizes memory addresses -> you need a leak
NX/DEP        the stack isn't executable -> can't put shellcode there (-> ROP, pwn-rop)
Stack Canary  a sentinel value before the return address -> detects overflow (must leak it)
PIE           the executable itself loads at a random address -> you need a leak
RELRO         protects the GOT (see pwn-got)
# see what a binary has
checksec --file=./binary

A modern exploit is usually: leak an address (bypass ASLR/PIE), bypass the canary, then redirect the flow (ROP because NX).

Tools of the trade

pwntools      the Python library to write exploits (interact, pack, ROP)
gdb + pwndbg/GEF   debug and view memory/registers with exploit helpers
checksec      see a binary's mitigations
ghidra / IDA / radare2   static analysis (reversing, see rev-*)
ropgadget / ropper   find gadgets for ROP
one_gadget    find a "magic gadget" that spawns a shell in libc

A pwn challenge workflow

1. checksec + reversing: understand what the binary does and its protections
2. identify the vulnerability (overflow, format string, heap...)
3. build a plan: do I need a leak? ROP? shellcode?
4. develop the exploit with pwntools (local first)
5. bypass mitigations (ASLR via leak, canary, NX via ROP)
6. get control of RIP -> shell / read the flag
7. launch against the remote

For the defense

Memory-safe languages (Rust, Go) where viable; in C/C++, safe practices (no unsafe functions like gets/strcpy).
Enable all mitigations: ASLR, NX, stack canaries, PIE, Full RELRO, FORTIFY_SOURCE.
Static/dynamic analysis (sanitizers: ASan, UBSan), fuzzing (AFL++) to find bugs before the attacker.
Validate sizes and bounds; don’t trust user-controlled input for sizes/indices.

Testing checklist

Understand process memory and the goal (control RIP)
checksec: see the binary’s mitigations
Identify the vulnerability class
Reverse the binary (ghidra/gdb) to understand the flow
Exploit plan per the protections (leak/ROP/shellcode)
Develop with pwntools (local)
Bypass ASLR/canary/NX/PIE
Control RIP → shell/flag → remote