Live data · Updated 18 Oct 2026
AI Crawler IndexWhich AI bots crawl a real website, and how often
AI crawlers made 34% of our bot traffic in the last 30 days. We log every crawler that visits mojodojo.io and sort it into AI training bots, AI search bots, assistants reading a page for a user, and search engines like Google and Bing. Each visit is checked against the IP addresses the bot's owner publishes, so anything pretending to be Googlebot or GPTBot is left out. We started checking IPs on 7 Oct 2026, so earlier visits are sorted by the name the bot gives itself.
AI crawlers
33.8%
29,479 of 87,339 counted bot hits.
Search engines
25.7%
AI bots made 1.31 hits for every search engine hit.
Live AI fetches
1,620
Pages an assistant like ChatGPT read on the spot because someone asked it a question.
Fake bots
7.6%
Of hits claiming to be a bot we can verify, the share that came from the wrong IP.
Trend
Crawler hits per day, fakes removed
- AI training
- AI search
- AI user fetch
- Search engines
- SEO tools
- Social previews
- Archive
- Other bots
Split
Who is crawling
| Category | Hits | Share |
|---|---|---|
| AI training | 23,360 | 26.7% |
| AI search | 4,499 | 5.2% |
| AI user fetch | 1,620 | 1.9% |
| Search engines | 22,443 | 25.7% |
| SEO tools | 15,708 | 18.0% |
| Social previews | 11,159 | 12.8% |
| Archive | 624 | 0.7% |
| Other bots | 7,926 | 9.1% |
Spoofed hits are excluded from every share on this page. 1,631 were removed in this period.
Leaderboard
Every bot, ranked
| Bot | Type | Hits | Per day | Pages | Hits / page | Verified | Fakes blocked |
|---|---|---|---|---|---|---|---|
| Meta-ExternalAgentMeta | AI training | 15,885 | 530 | 2,570 | 6.2 | — | — |
| facebookexternalhitMeta | Social previews | 9,368 | 312 | 112 | 83.6 | — | — |
| AhrefsBotAhrefs | SEO tools | 8,982 | 299 | 3,114 | 2.9 | 100% | — |
| Other | Other bots | 7,926 | 264 | 1,623 | 4.9 | — | — |
| PetalBotHuawei | Search engines | 6,881 | 229 | 3,471 | 2.0 | 100% | — |
| GooglebotGoogle | Search engines | 6,197 | 207 | 1,409 | 4.4 | 92% | 180 |
| BingbotMicrosoft | Search engines | 5,641 | 188 | 2,610 | 2.2 | 96% | 67 |
| PerplexityBotPerplexity | AI search | 3,448 | 115 | 1,356 | 2.5 | 92% | 131 |
| DotBotMoz | SEO tools | 3,285 | 110 | 2,859 | 1.1 | — | — |
| ClaudeBotAnthropic | AI training | 3,279 | 109 | 1,552 | 2.1 | 91% | 95 |
| AmazonbotAmazon | AI training | 2,579 | 86 | 917 | 2.8 | 90% | 82 |
| SemrushBotSemrush | SEO tools | 2,288 | 76 | 1,417 | 1.6 | 97% | 18 |
| ApplebotApple | Search engines | 1,701 | 57 | 756 | 2.3 | 80% | 120 |
| LinkedInBotLinkedIn | Social previews | 1,694 | 57 | 22 | 77.0 | — | — |
| MJ12botMajestic | SEO tools | 1,144 | 38 | 803 | 1.4 | — | — |
| GPTBotOpenAI | AI training | 1,131 | 38 | 925 | 1.2 | 77% | 126 |
| SogouSogou | Search engines | 1,019 | 34 | 205 | 5.0 | — | — |
| ChatGPT-UserOpenAI | AI user fetch | 929 | 31 | 278 | 3.3 | 75% | 107 |
| OAI-SearchBotOpenAI | AI search | 927 | 31 | 580 | 1.6 | 77% | 106 |
| Internet ArchiveInternet Archive | Archive | 624 | 21 | 387 | 1.6 | — | — |
| YandexBotYandex | Search engines | 414 | 14 | 157 | 2.6 | 100% | — |
| DuckAssistBotDuckDuckGo | AI user fetch | 216 | 7 | 149 | 1.4 | — | — |
| BaiduspiderBaidu | Search engines | 212 | 7 | 136 | 1.6 | 19% | 100 |
| cohere-aiCohere | AI training | 206 | 7 | 143 | 1.4 | — | — |
| Claude-UserAnthropic | AI user fetch | 199 | 7 | 116 | 1.7 | 34% | 151 |
| GoogleOtherGoogle | Search engines | 191 | 6 | 141 | 1.4 | 100% | — |
| BytespiderByteDance | AI training | 187 | 6 | 133 | 1.4 | — | — |
| DuckDuckBotDuckDuckGo | Search engines | 187 | 6 | 26 | 7.2 | 100% | — |
| Meta-ExternalFetcherMeta | AI user fetch | 160 | 5 | 118 | 1.4 | — | — |
| Claude-SearchBotAnthropic | AI search | 124 | 4 | 98 | 1.3 | 0% | 118 |
| Perplexity-UserPerplexity | AI user fetch | 116 | 4 | 93 | 1.2 | 0% | 101 |
| CCBotCommon Crawl | AI training | 93 | 3 | 74 | 1.3 | 0% | 129 |
| TwitterbotX | Social previews | 30 | 1 | 9 | 3.3 | — | — |
| PinterestbotPinterest | Social previews | 25 | 1 | 5 | 5.0 | 100% | — |
| SlackbotSlack | Social previews | 25 | 1 | 3 | 8.3 | — | — |
| DiscordbotDiscord | Social previews | 15 | 1 | 4 | 3.8 | — | — |
| Screaming FrogScreaming Frog | SEO tools | 9 | 0 | 3 | 3.0 | — | — |
| TelegramBotTelegram | Social previews | 1 | 0 | 1 | 1.0 | — | — |
| WhatsAppMeta | Social previews | 1 | 0 | 1 | 1.0 | — | — |
Bot awards
The standouts of the last 30 days
Most aggressive
Meta-ExternalAgent
530 hits a day · Meta · AI training
Most hits per day
Most polite
DotBot
1.1 hits per page · Moz · SEO tools
Top AI bot
GPTBot (OpenAI) · 1.2 hits per page
Fewest hits per page: visits once and moves on
Most obsessive
facebookexternalhit
83.6 hits per page · Meta · Social previews
Top AI bot
Meta-ExternalAgent (Meta) · 6.2 hits per page
Most hits per page: keeps fetching the same few
Most thorough
PetalBot
3,471 pages · Huawei · Search engines
Top AI bot
Meta-ExternalAgent (Meta) · 2,570 pages
Most different pages visited
Rarest visitor
CCBot
3 days, 93 hits · Common Crawl · AI training
Fewest days seen (at least 10 hits)
Most impersonated
Googlebot
180 fake hits · Google · Search engines
Top AI bot
Claude-User (Anthropic) · 151 fake hits
Most fake hits caught by the IP check
A bot needs at least 100 hits in the period to win (10 for rarest visitor). Where an AI bot would win its own category, it is listed underneath.
Methodology
How the numbers are made
Source. Every page request to mojodojo.io whose user agent looks like a crawler, logged at the web server with the visitor's IP. One site, so read the numbers as a case study rather than a census. Tracking since 6 June 2026. IPs were recorded from 7 Oct 2026; hits before that are classified by user agent only and counted as unverified.
Categories. AI training bots collect pages for model training (GPTBot, ClaudeBot, CCBot). AI search bots build the index behind AI answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). AI user fetches are live reads on behalf of a person (ChatGPT-User, Claude-User, Perplexity-User).
Verification. A hit counts as verified when its IP sits inside the ranges the operator publishes (OpenAI, Anthropic, Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs), or passes forward-confirmed reverse DNS. Ranges refresh daily. A hit that fails the check is marked spoofed and left out. Bots whose operator publishes no method are counted as unverifiable.
Cite this. Mojo Dojo, “AI Crawler Index”, mojodojo.io/ai-crawler-index, accessed 18 Oct 2026. Free to reuse under CC BY 4.0 with a link back.
| Bot | Type | Check |
|---|---|---|
| ChatGPT-User | AI user fetch | Published IP ranges |
| Claude-User | AI user fetch | Published IP ranges |
| Perplexity-User | AI user fetch | Published IP ranges |
| MistralAI-User | AI user fetch | Not verifiable |
| DuckAssistBot | AI user fetch | Not verifiable |
| Meta-ExternalFetcher | AI user fetch | Not verifiable |
| OAI-SearchBot | AI search | Published IP ranges |
| Claude-SearchBot | AI search | Published IP ranges |
| PerplexityBot | AI search | Published IP ranges |
| GPTBot | AI training | Published IP ranges |
| ClaudeBot | AI training | Published IP ranges |
| CCBot | AI training | Published IP ranges |
| Meta-ExternalAgent | AI training | Not verifiable |
| FacebookBot | AI training | Not verifiable |
| Bytespider | AI training | Not verifiable |
| Amazonbot | AI training | Reverse DNS |
| cohere-ai | AI training | Not verifiable |
| Diffbot | AI training | Not verifiable |
| Googlebot | Search engines | IP ranges + reverse DNS |
| GoogleOther | Search engines | IP ranges + reverse DNS |
| Bingbot | Search engines | IP ranges + reverse DNS |
| DuckDuckBot | Search engines | Published IP ranges |
| Applebot | Search engines | IP ranges + reverse DNS |
| YandexBot | Search engines | Reverse DNS |
| Baiduspider | Search engines | Reverse DNS |
| PetalBot | Search engines | Reverse DNS |
| Sogou | Search engines | Not verifiable |
| AhrefsBot | SEO tools | Published IP ranges |
| SemrushBot | SEO tools | Reverse DNS |
| MJ12bot | SEO tools | Not verifiable |
| DotBot | SEO tools | Not verifiable |
| Screaming Frog | SEO tools | Not verifiable |
| facebookexternalhit | Social previews | Not verifiable |
| Twitterbot | Social previews | Not verifiable |
| LinkedInBot | Social previews | Not verifiable |
| Pinterestbot | Social previews | Reverse DNS |
| Slackbot | Social previews | Not verifiable |
| Discordbot | Social previews | Not verifiable |
| TelegramBot | Social previews | Not verifiable |
| Social previews | Not verifiable | |
| Internet Archive | Archive | Not verifiable |