Content Discovery

A website shows only what it wants you to see; content discovery finds what it doesn’t link: hidden directories, admin panels, backup files, API endpoints, old versions, .git/, config files. It’s active enumeration over HTTP: you try dictionary paths against the server and watch what responds. What isn’t linked is usually the most interesting.

The principle

You request candidate paths (/admin, /backup.zip, /api/v1) and classify by the response code: 200 (exists), 301/302 (redirects), 403 (exists but forbidden — interesting!), 404 (doesn’t exist). The 403 is gold: it confirms the resource is there.

Directory and file fuzzing

# ffuf (fast, flexible)
ffuf -w wordlist.txt -u https://target/FUZZ -mc 200,301,302,403 -o out.json
# feroxbuster (recursive by default)
feroxbuster -u https://target -w wordlist.txt -x php,bak,zip,old,txt
# gobuster
gobuster dir -u https://target -w wordlist.txt -x php,html,bak

Filter noise by size/words when the 404 returns 200 (soft-404): -fs <size> / -fw <words>.

Juicy extensions and files

# backups and sources
.bak .old .zip .tar.gz .sql .swp index.php~ config.php.bak
# version control and configs exposed
/.git/  /.svn/  /.env  /.htaccess  /web.config  /wp-config.php.bak
# discover .git/ and reconstruct code (see fund-git)
curl -s https://target/.git/HEAD

Wordlists (the key to results)

SecLists:  Discovery/Web-Content/   (raft-*, directory-list-*, common.txt)
# technology-specific (if you fingerprinted)
Discovery/Web-Content/CMS/  (wordpress, joomla...)
# generated from the site itself
cewl https://target -w custom.txt   # the site's words as a dictionary

Endpoints and paths from the site itself

Not everything is brute force; the site already reveals paths:

# extract endpoints from JavaScript (SPA, APIs)
katana -u https://target -jc        # crawling + JS
# linkfinder / gau / waymore: historical and JS URLs
gau target.com ; waymore -i target.com   # (see recon-wayback)
# robots.txt and sitemap.xml reveal paths they "didn't want" indexed
curl -s https://target/robots.txt https://target/sitemap.xml

APIs

# exposed documentation
/swagger.json /openapi.json /api-docs /graphql
# API versions and paths by fuzzing with an API wordlist
ffuf -w api-wordlist.txt -u https://target/api/FUZZ

For the defense

Reduce findings: don’t leave backups/configs in the webroot, block access to .git//.env/.svn, remove old panels and versions, return consistent 404s (don’t leak existence via 403/timing), and watch logs for fuzzing (bursts of 404/403). A WAF with rate-limit slows mass brute force.

Testing checklist

Directory fuzzing with a good wordlist (SecLists)
File fuzzing by extension (bak, zip, sql, env, old)
Filter soft-404 by size/words
.git/, .env, .svn, exposed configs
robots.txt and sitemap.xml reviewed
Endpoints extracted from JS (katana/linkfinder)
Historical URLs (gau/waymore)
API documentation and paths (swagger/graphql)