Content Discovery in Cybersecurity Pentesting

Content discovery is a critical phase in penetration testing—uncovering hidden files, directories, and APIs that could lead to vulnerabilities. While manual checks with Python and curl work, automated tools speed up the process.

In this updated post, we’ll explore additional tools like DirBuster, Gobuster, Burp Suite, and Wfuzz, along with examples targeting juice-shop.herokuapp.com.

1. Manual Content Discovery with Python (Recap)

A quick Python script to check common endpoints:

import requests

target = "https://juice-shop.herokuapp.com"
paths = ["/admin", "/login", "/api", "/backup", "/config"]

for path in paths:
    url = target + path
    try:
        res = requests.get(url, timeout=5)
        print(f"{url} → Status: {res.status_code}")
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")

Output:

https://juice-shop.herokuapp.com/admin → Status: 404
https://juice-shop.herokuapp.com/login → Status: 200
https://juice-shop.herokuapp.com/api → Status: 200

2. Fast Scanning with ffuf (Recap)

A powerful web fuzzer for directory/file discovery:

ffuf -w /usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
     -u https://juice-shop.herokuapp.com/FUZZ \
     -mc 200,401,403 -t 50

• -w: Wordlist (e.g., SecLists).
• -u: Target with FUZZ placeholder.
• -mc: Filter by status codes.
• -t: Threads (speed up scanning).

Output:

api                     [Status: 200, Size: 2]
login                   [Status: 200, Size: 1024]

3. New Tools for Content Discovery

A. DirBuster (GUI Tool)

A Java-based directory brute-forcer with a GUI.

Steps:

1. Download DirBuster.
2. Load a wordlist (e.g., directory-list-2.3-medium.txt).
3. Set target: https://juice-shop.herokuapp.com.
4. Start scanning.

Result: Finds /api, /login, /assets, etc.

B. Gobuster (CLI Alternative to DirBuster)

A faster, command-line alternative to DirBuster.

Example:

gobuster dir -u https://juice-shop.herokuapp.com \
            -w /usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
            -t 50 -x txt,json,php

• -x: Extensions to check (e.g., .txt, .json).
• -t: Threads (higher = faster but noisier).

Output:

/api (Status: 200)
/login (Status: 200)
/assets (Status: 302)

C. Burp Suite (Professional Tool)

A commercial web proxy with Intruder for content discovery.

Steps:

1. Configure Burp as a proxy.
2. Go to Intruder → Positions and set https://juice-shop.herokuapp.com/§FUZZ§.
3. Load a wordlist in Payloads.
4. Start the attack.

Result: Identifies /api, /login, /admin (if accessible).

D. Wfuzz (Advanced Fuzzing)

A flexible tool for parameter and directory discovery.

Example:

wfuzz -c -z file,/usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
      -u https://juice-shop.herokuapp.com/FUZZ --hc 404

• -c: Colorful output.
• --hc 404: Hide 404 responses.

Output:

000001:  200 → /api
000002:  200 → /login

E. Nikto (Vulnerability Scanner with Discovery)

While primarily a vulnerability scanner, Nikto checks for common files:

nikto -h https://juice-shop.herokuapp.com

Output:

+ /login.php (Found)
+ /api/v1 (Found)

4. Comparing Tools

Final Recommendations

• For beginners: Start with Python + ffuf.
• For speed: Use Gobuster or ffuf.
• For deep testing: Burp Suite + Wfuzz.
• For quick checks: curl or Nikto.

Happy scanning! 🔍🚀