Anti-debugging and Anti-analysis

Software that doesn’t want to be analyzed —malware, license protections, DRM, anti-cheat— incorporates techniques to detect and obstruct the analyst: check whether a debugger is attached, whether it runs in a virtual machine or sandbox, whether it’s being instrumented, or whether the binary itself has been modified. On detection, it changes behavior, exits, or runs decoy logic. For the reverser it’s a cat-and-mouse game: recognize these defenses and bypass them.

Anti-debugging (detect the debugger)

# Linux
ptrace(PTRACE_TRACEME)      # a process can only be traced once;
                            # if the program traces itself, a debugger can't attach
/proc/self/status           # the TracerPid field != 0 indicates an attached debugger
# Windows
IsDebuggerPresent() / CheckRemoteDebuggerPresent() / PEB->BeingDebugged
NtQueryInformationProcess   (ProcessDebugPort, etc.)
# generic
timing checks               # code under a debugger is MUCH slower (rdtsc, GetTickCount)
breakpoint detection        # look for 0xCC (int3) bytes in its own code
signals/exceptions          # handle SIGTRAP in a way that exposes the debugger

Anti-VM / anti-sandbox (detect the analysis environment)

# virtualization artifacts
CPU instructions/registers (cpuid hypervisor bit), VMware/VBox MAC addresses
characteristic VM/sandbox files/processes/registry keys
# sandbox detection
little user activity, low uptime, few cores/RAM
long sleeps (sandboxes limit analysis time -> sleep to evade)
no human interaction (no mouse movement)

Anti-tampering (detect modification)

# checksums/hashes of its own code -> if someone patches the binary, it doesn't match
# runtime integrity checks
# self-modifying / encrypted code (static doesn't see the real code, see rev-static)

Obfuscation (hinder understanding)

control flow flattening     # flatten the flow so the graph is unreadable
opaque predicates           # always-true/false conditions that confuse
string encryption           # strings decrypted only at runtime
dead code / junk            # junk instructions that do nothing
code virtualization         # a custom interpreter runs custom bytecode (very hard)
packers/protectors          # VMProtect, Themida, UPX... (see rev-dynamic to unpack)

How the analyst bypasses them

# anti-debug
- patch the check (NOP over the if, force the value with GDB: set $rax=0)
- LD_PRELOAD a fake ptrace (Linux) that always returns success
- plugins: ScyllaHide (Windows), hooks that hide the debugger
# anti-VM
- more "realistic" VMs (hide artifacts), bare-metal for very evasive samples
- patch the VM checks
# anti-sandbox (sleeps)
- speed up time / patch sleeps / hook GetTickCount
# obfuscation
- dynamic (rev-dynamic): run and dump what's decrypted; deobfuscators; angr
# anti-tampering
- also patch the integrity check (or before the rest)

The principle: these protections raise the cost of analysis, they don’t prevent it. Almost all reduce to a check that, once located, can be patched or forced.

For the defense (developer)

# combine several layers (a single one is easily patched): anti-debug + anti-VM + integrity + obfuscation
# commercial protectors (VMProtect/Themida) to raise the cost a lot
# remember: NONE fully protects against a determined analyst ->
#   for malware, detection (EDR/sandbox) is the answer; for IP, delay the pirate

Testing checklist

Detect anti-debug (ptrace/IsDebuggerPresent/timing)
Bypass it (patch/force/LD_PRELOAD/ScyllaHide)
Detect anti-VM/anti-sandbox (artifacts, sleeps)
Evade (realistic VM / patch / speed up time)
Detect anti-tampering (checksums) and patch it
Identify obfuscation/packing and fight it (dynamic/angr)
Combine static + dynamic for protected code
Document the protections found