Settings

Theme

One in Five Websites Blocks an AI Crawler

ciphercue.com

2 points by adulion · 3 comments

Reader

2 threads
Bender

robots.txt does not block bots, it is just a suggestion. There are no legal teeth behind robots.txt and AI training bots can and do just ignore it. There are a myriad of methods to actually block bots, here is just one of them (and it will block Google's crawler as their http library is out of date). I use the ban variable to put them in a different access log and dump their IP into NFTables ipsets. This is mostly for small sites that need not be crawled or archives that get abused by poorly written or rude bots.

    # nginx rule to block some crawlers/bots, permit HTTP/2.0 and HTTP/3.0, block all other protocols.

    if ($server_protocol = HTTP/2.0) { set $ok 1; }
    if ($http3 != "") { set $ok 1; }
    if ($ok != 1) { set $ban 1; return 444 ''; }
neerajjain22

If the content is not your core offering and your systems are not abused by cyberattacks, then it makes zero sense to block AI crawlers.

Ai agents are your new buyers.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection