Content Discovery in Cybersecurity Pentesting

Content discovery is a critical phase in penetration testing—uncovering hidden files, directories, and APIs that could lead to vulnerabilities. While manual checks with Python and curl work, automated tools speed up the process.

In this updated post, we’ll explore additional tools like DirBuster, Gobuster, Burp Suite, and Wfuzz, along with examples targeting juice-shop.herokuapp.com.

---

1. Manual Content Discovery with Python (Recap)

A quick Python script to check common endpoints:

import requests

target = "https://juice-shop.herokuapp.com"
paths = ["/admin", "/login", "/api", "/backup", "/config"]

for path in paths:
    url = target + path
    try:
        res = requests.get(url, timeout=5)
        print(f"{url} → Status: {res.status_code}")
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")

Output:

https://juice-shop.herokuapp.com/admin → Status: 404
https://juice-shop.herokuapp.com/login → Status: 200
https://juice-shop.herokuapp.com/api → Status: 200

---

2. Fast Scanning with ffuf (Recap)

A powerful web fuzzer for directory/file discovery:

ffuf -w /usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
     -u https://juice-shop.herokuapp.com/FUZZ \
     -mc 200,401,403 -t 50

• -w: Wordlist (e.g., SecLists).
• -u: Target with FUZZ placeholder.
• -mc: Filter by status codes.
• -t: Threads (speed up scanning).

Output:

api                     [Status: 200, Size: 2]
login                   [Status: 200, Size: 1024]

---

3. New Tools for Content Discovery

A. DirBuster (GUI Tool)

A Java-based directory brute-forcer with a GUI.

Steps:

1. Download DirBuster.
2. Load a wordlist (e.g., directory-list-2.3-medium.txt).
3. Set target: https://juice-shop.herokuapp.com.
4. Start scanning.

Result: Finds /api, /login, /assets, etc.

---

B. Gobuster (CLI Alternative to DirBuster)

A faster, command-line alternative to DirBuster.

Example:

gobuster dir -u https://juice-shop.herokuapp.com \
            -w /usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
            -t 50 -x txt,json,php

• -x: Extensions to check (e.g., .txt, .json).
• -t: Threads (higher = faster but noisier).

Output:

/api (Status: 200)
/login (Status: 200)
/assets (Status: 302)

---

C. Burp Suite (Professional Tool)

A commercial web proxy with Intruder for content discovery.

Steps:

1. Configure Burp as a proxy.
2. Go to Intruder → Positions and set https://juice-shop.herokuapp.com/§FUZZ§.
3. Load a wordlist in Payloads.
4. Start the attack.

Result: Identifies /api, /login, /admin (if accessible).

---

D. Wfuzz (Advanced Fuzzing)

A flexible tool for parameter and directory discovery.

Example:

wfuzz -c -z file,/usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
      -u https://juice-shop.herokuapp.com/FUZZ --hc 404

• -c: Colorful output.
• --hc 404: Hide 404 responses.

Output:

000001:  200 → /api
000002:  200 → /login

---

E. Nikto (Vulnerability Scanner with Discovery)

While primarily a vulnerability scanner, Nikto checks for common files:

nikto -h https://juice-shop.herokuapp.com

Output:

+ /login.php (Found)
+ /api/v1 (Found)

---

4. Comparing Tools

Tool	Type	Speed	Ease of Use	Best For
Python	Manual	Slow	High	Quick checks
ffuf	CLI Fuzzer	Fast	Medium	Large-scale scans
Gobuster	CLI Brute	Fast	High	Directory/file discovery
DirBuster	GUI Brute	Medium	Low	Beginners
Burp	Pro Proxy	Medium	Low	Advanced testing
Wfuzz	Advanced	Fast	Medium	Parameter/API fuzzing

---

Final Recommendations

• For beginners: Start with Python + ffuf.
• For speed: Use Gobuster or ffuf.
• For deep testing: Burp Suite + Wfuzz.
• For quick checks: curl or Nikto.

---

Happy scanning! 🔍🚀