GitHub - ryanbr/network-scanner: A network scanner to output in various adblock formats

A Puppeteer-based tool for scanning websites to find third-party (or optionally first-party) network requests matching specified patterns, and generate Adblock-formatted rules.

Features

• Scan websites and detect matching third-party or first-party resources
• Output Adblock-formatted blocking rules
• Support for multiple filters per site
• Grouped titles (! ) before site matches
• Ignore unwanted domains (global and per-site)
• Block unwanted domains during scan (simulate adblock)
• Support Chrome, Firefox, Safari user agents (desktop or mobile)
• Advanced fingerprint spoofing and referrer header simulation
• Delay, timeout, reload options per site
• Verbose and debug modes
• Dump matched full URLs into matched_urls.log
• Save output in normal Adblock format or localhost (127.0.0.1/0.0.0.0)
• Subdomain handling (collapse to root or full subdomain)
• Optionally match only first-party, third-party, or both
• Enhanced redirect handling with JavaScript and meta refresh detection

Command Line Arguments

Output Options

Output Format Options

General Options

Validation Options

config.json Format

Example:

{
  "ignoreDomains": [
    "googleapis.com",
    "googletagmanager.com"
  ],
  "sites": [
    {
      "url": "https://example.com/",
      "userAgent": "chrome",
      "filterRegex": "ads|analytics",
      "resourceTypes": ["script", "xhr", "image"],
      "reload": 2,
      "delay": 5000,
      "timeout": 30000,
      "verbose": 1,
      "debug": 1,
      "interact": true,
      "fingerprint_protection": "random",
      "referrer_headers": {
        "mode": "random_search",
        "search_terms": ["example reviews", "best deals"]
      },
      "custom_headers": {
        "X-Custom-Header": "value"
      },
      "firstParty": 0,
      "thirdParty": 1,
      "subDomains": 0,
      "blocked": [
        "googletagmanager.com",
        ".*tracking.*"
      ]
    }
  ]
}

config.json Field Table

Basic Configuration

Window cleanup modes: false (disabled), true (conservative - closes obvious leftovers), "all" (aggressive - closes all content pages). Both active modes preserve the main Puppeteer window and wait 16 seconds before cleanup to avoid interfering with active operations.

Redirect Handling Options

When a page redirects to a new domain, first-party/third-party detection is based on the final redirected domain, and all intermediate redirect domains (like bit.ly, t.co) are automatically excluded from the generated rules.

Advanced Stealth & Fingerprinting

Referrer Header Options

Simple formats:

"referrer_headers": "https://google.com/search?q=example"
"referrer_headers": ["url1", "url2"]

Smart modes:

"referrer_headers": {"mode": "random_search", "search_terms": ["reviews"]}
"referrer_headers": {"mode": "social_media"}
"referrer_headers": {"mode": "direct_navigation"}
"referrer_headers": {"mode": "custom", "custom": ["https://news.ycombinator.com/"]}

Protection Bypassing

WHOIS/DNS Analysis Options

Content Analysis Options

Advanced Browser Options

Global Configuration Options

These options go at the root level of your config.json:

Special Characters in searchstring

The searchstring parameter supports all characters including special symbols. Only double quotes need JSON escaping:

{
  "searchstring": [
    ")}return n}function N(n,e,r){try{\"function\"==typeof",
    "addEventListener(\"click\",function(){",
    "{\"status\":\"success\",\"data\":[",
    "console.log('Debug: ' + value);",
    "`API endpoint: ${baseUrl}/users`",
    "@media screen and (max-width: 768px)",
    "if(e&&e.preventDefault){e.preventDefault()}",
    "__webpack_require__(/*! ./module */ \"./src/module.js\")",
    "console.log('Hello world')",
    "#header { background-color: #ff0000; }",
    "$(document).ready(function() {",
    "completion: 85% @ $1,500 budget",
    "SELECT * FROM users WHERE id = *",
    "regex: ^[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}$",
    "typeof window !== 'undefined'"
  ]
}

Character escaping rules:

• " becomes \" (required in JSON)
• \ becomes \\ (if searching for literal backslashes)
• All other characters are used literally: ' ` @ # $ % * ^ [ ] { } ( ) ; = ! ? :

Usage Examples

Basic Scanning

# Scan with default config and output to console
node nwss.js

# Scan and save rules to file
node nwss.js -o blocklist.txt

# Append new rules to existing file
node nwss.js --append -o blocklist.txt

# Clean existing rules and append new ones
node nwss.js --clean-rules --append -o blocklist.txt

Advanced Options

# Debug mode with URL dumping and colored output
node nwss.js --debug --dumpurls --color -o rules.txt

# Dry run to see what would be matched
node nwss.js --dry-run --debug

# Validate configuration before running
node nwss.js --validate-config

# Clean rule files
node nwss.js --clean-rules existing_rules.txt

# Maximum stealth scanning
node nwss.js --debug --color -o stealth_rules.txt

Performance Tuning

# High-performance scanning with custom concurrency
node nwss.js --max-concurrent 12 --cleanup-interval 300 -o rules.txt

Stealth Configuration Examples

Memory Management with Window Cleanup

{
  "url": [
    "https://popup-heavy-site1.com",
    "https://popup-heavy-site2.com", 
    "https://popup-heavy-site3.com"
  ],
  "filterRegex": "\\.(space|website|tech)\\b",
  "window_cleanup": "all",
  "interact": true,
  "reload": 2,
  "resourceTypes": ["script", "fetch"],
  "comments": "Aggressive cleanup for sites that open many popups"
}

Conservative Memory Management

{
  "url": "https://complex-site.com",
  "filterRegex": "analytics|tracking",
  "window_cleanup": true,
  "interact": true,
  "delay": 8000,
  "reload": 3,
  "comments": [
    "Conservative cleanup preserves potentially active content",
    "Good for sites with complex iframe structures"
  ]
}

E-commerce Site Scanning

{
  "url": "https://shopping-site.com",
  "userAgent": "chrome",
  "fingerprint_protection": "random",
  "referrer_headers": {
    "mode": "random_search",
    "search_terms": ["product reviews", "best deals", "price comparison"]
  },
  "interact": true,
  "delay": 6000,
  "filterRegex": "analytics|tracking|ads"
}

News Site Analysis

{
  "url": "https://news-site.com",
  "userAgent": "firefox",
  "fingerprint_protection": true,
  "referrer_headers": {"mode": "social_media"},
  "custom_headers": {
    "Accept-Language": "en-US,en;q=0.9"
  },
  "filterRegex": "doubleclick|googletagmanager"
}

Tech Blog with Custom Referrers

{
  "url": "https://tech-blog.com",
  "fingerprint_protection": "random",
  "referrer_headers": {
    "mode": "custom",
    "custom": [
      "https://news.ycombinator.com/",
      "https://www.reddit.com/r/programming/",
      "https://lobste.rs/"
    ]
  }
}

Memory Management

The scanner includes intelligent window management to prevent memory accumulation during long scans:

• Conservative cleanup (window_cleanup: true): Selectively closes pages that appear to be leftovers from previous scans
• Aggressive cleanup (window_cleanup: "all"): Closes all content pages from previous operations for maximum memory recovery
• Main window preservation: Both modes always preserve the main Puppeteer browser window to maintain stability
• Popup window handling: Automatically detects and closes popup windows created by previous site scans
• Timing protection: 16-second delay ensures no active operations are interrupted during cleanup
• Active page protection: Never affects pages currently being processed by concurrent scanning operations
• Memory reporting: Reports estimated memory freed from closed windows for performance monitoring

Use aggressive cleanup for sites that open many popups or when processing large numbers of URLs. Use conservative cleanup when you want to preserve potentially active content but still free obvious leftovers.

INSTALL

(Ubuntu as example). NOTE: Use Chrome and not Chromium for best compatibility.

Add Google's signing key

wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/googlechrome-linux-keyring.gpg

Add Google Chrome repository

echo "deb [arch=amd64 signed-by=/usr/share/keyrings/googlechrome-linux-keyring.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list

Update and install

sudo apt update
sudo apt install google-chrome-stable

dig & whois (needed for network checks)

sudo apt install bind9-dnsutils whois

Notes

• If both firstParty: 0 and thirdParty: 0 are set for a site, it will be skipped.
• ignoreDomains applies globally across all sites.
• ignoreDomains supports wildcards (e.g., *.ads.com matches tracker.ads.com)
• Blocking (blocked) can match full domains or regex.
• If a site's blocked field is missing, no extra blocking is applied.
• --clean-rules with --append will clean existing files first, then append new rules
• --remove-dupes works with all output modes and removes duplicates from final output
• Validation tools help ensure rule files are properly formatted before use
• --remove-tempfiles removes Chrome/Puppeteer temporary files before exiting, avoids disk space issues
• For maximum stealth, combine fingerprint_protection: "random" with appropriate referrer_headers modes
• User agents are automatically updated to latest versions (Chrome 131, Firefox 133, Safari 18.2)
• Referrer headers work independently from fingerprint protection - use both for best results