GitHub - AdrianKrebs/design-slop-cop: A tool to score websites for AI design patterns.

A tool to score any website for common AI design patterns. Live: www.slopcop.adriankrebs.ch — paste a URL, get a scored verdict. Basis for the blog post and the HN discussion "Scoring Show HN submissions for AI design patterns".

The tool loads each site in a headless browser, analyzes the DOM, and reports which of 16 deterministic AI design patterns are found. Manual verification across ~150 labeled sites suggests ~5–10% false positives. Still, take it with a grain of salt :)

Install

git clone https://github.com/AdrianKrebs/design-slop-cop
cd design-slop-cop && npm install

Requires Node 18+.

Check a single URL

A clean site (no AI design patterns triggered):

$ node check.js https://news.ycombinator.com/
https://news.ycombinator.com/
Low slop · score 0/100 · 0/16 patterns

A heavy one (7 patterns triggered):

$ node check.js https://engagemii.com/aeo
https://engagemii.com/aeo
High slop · score 44/100 · 7/16 patterns

Triggered:
  • Vibe purple
  • Gradients
  • Perma dark
  • 1·2·3 steps
  • Stat banner
  • Headline badge
  • FAQ

Add --json for machine-readable output:

$ node check.js https://engagemii.com/aeo --json
{
  "url": "https://engagemii.com/aeo",
  "score": 44,
  "tierLabel": "Heavy",
  "patternsFlagged": 7,
  "patternsTotal": 16,
  "patterns": [
    { "id": "purple_accent", "label": "Vibe purple", "triggered": true, "evidence": {...} },
    ...
  ]
}

To update an existing clone, git pull inside the directory.

Run the web frontend

The same UI as the live site at www.slopcop.adriankrebs.ch — paste a URL, get a scored verdict and screenshot:

npm start          # → http://localhost:8080

The first scan is slow (it warms up a headless browser); subsequent scans reuse it. /show serves the browsable Show HN gallery. To deploy it publicly, see DEPLOY.md.

Run the full corpus

For batch analysis (e.g. the bundled urls.txt with ~1000 Show HN posts) — same install as above, then:

npm run analyze                      # sequential, ~100 min for 1k URLs
node src/run.js --concurrency=4      # 4 parallel, ~25 min, ~600 MB RAM
node src/run.js --skip-existing      # only fetch URLs not yet cached

Results go to results/:

• results/raw/<slug>.json — per-URL signals + score
• results/screenshots/<slug>.png — full-page screenshot
• results/all-results.json — all scored entries combined

Patterns

The full rule for each pattern lives in src/patterns/<id>.js.

Score

score      = round(100 × patternsFlagged / patternsTotal)
slop level = ≥4 High · 2–3 Medium · 0–1 Low

Tools

npm run scan      # → http://localhost:7788  minimal dev scanner (see "Run the web frontend" for the full UI)
npm run label     # → http://localhost:7777  label sites for ground truth
npm run eval      # precision / recall vs dataset/labels.jsonl (165 labels shipped)
npm run report    # generate results/index.html — browsable tier-filtered grid
npm run fetch     # pull the latest 100 Show HN URLs into urls.txt

The scan UI launches a real browser per URL, so the first scan after starting the server is slower. The label UI lets you mark each pattern present / not_present / skip; saves are appended to dataset/labels.jsonl. The eval script compares the detector's verdict against those labels.

Adding a pattern

Spotted a tell we miss? Each pattern is one self-contained file. To add one:

1. Copy the template: cp src/patterns/_template.js src/patterns/<your-id>.js. It documents the full export shape, everything available in ctx, and the one gotcha (below).
2. Fill in the exports: { id, label, shortLabel, description, category, thresholds, extract(ctx), score(signal, T) }.
   • extract(ctx) runs in the browser (it's serialized with Function.prototype.toString()), so keep it self-contained: reference only ctx.* — no imports, closures, or outer variables.
   • score(signal, T) runs in Node and returns { triggered: true|false, evidence: ... }.
3. Register it: import the file in src/patterns/index.js and append it to the PATTERNS array (array order = display order).
4. Debug it: node check.js <a-site-that-should-trigger-it> --pattern=<your-id> prints the raw signal your extract returned and the verdict your score returned. Confirm it fires there and stays quiet on clean sites.
5. Check accuracy: npm run eval to make sure precision/recall didn't regress, then open a PR.

License

MIT.