Overview
Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
Bot vs. Human Traffic
↓ 3%
Compared to the previous 90 days
The amount of visits from known agents vs. humans
Agentrification
↑ 7%
Compared to the previous 90 days
The percentage of bot traffic that's AI-related
AI Chat Referral Volume
↑ 20%
Compared to the previous 90 days
The percentage of human website visits that come from AI chat
Robots.txt Effectiveness
How often bots follow robots.txt rules
Traffic by Agent Type
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
Automated Agent
Automated Agent
Automates browser interactions programmatically without direct human supervision
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
Fetcher
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
Security Scanner
Security Scanner
Scans websites for security vulnerabilities, threats, and configuration weaknesses
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
Uncategorized
Uncategorized
Not yet assigned a type
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Hover over each agent type for more information about what they do
Top Agent Types
Agent types with the most activity
Top Visiting Agents
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
9.2%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
7.2%
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
6.8%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
6.6%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.4%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.8%
FTCH
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
2.8%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
2.6%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
1.9%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.8%
Agents with the most activity
Top Operators
Operators with the most activity
AI Training Activity
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers.
Traffic by Agent Type
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI training activity by agent type over time
Top Visited Website Categories
Computers and Electronics
6.5%
Business and Industrial
5.9%
Internet and Telecom
5.5%
Website categories with most activity
Top Agents
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
25.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
17.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
16.9%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
7.0%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
6.7%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
6.6%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
4.2%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
3.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.3%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
1.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.8%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.5%
Agents doing the most AI training
Top Operators
Operators doing the most AI training
AI Fetching Activity
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents.
Traffic by Agent Type
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI fetching activity by agent type over time
Top Visited Website Categories
Computers and Electronics
1.9%
Internet and Telecom
1.6%
Travel and Transportation
1.6%
Business and Industrial
1.5%
Website categories with most activity
Top Agents
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
84.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
5.6%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
4.8%
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
2.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.6%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.2%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.1%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.1%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Agents doing the most AI fetching
Top Operators
Operators doing the most AI fetching
AI Search Indexing Activity
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
Traffic by Agent Type
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
AI search indexing activity by agent type over time
Top Visited Website Categories
Travel and Transportation
7.1%
Books and Literature
5.6%
Arts and Entertainment
5.0%
Computers and Electronics
4.7%
Website categories with most activity
Top Agents
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
26.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
15.7%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
13.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
12.7%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
12.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.6%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.7%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
1.8%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.4%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.3%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Agents doing the most AI search indexing
Top Operators
Operators doing the most AI search indexing
AI Browsing Activity
These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents.
Average Session Duration
↑ 63%
Compared to the previous 90 days
The average duration of a session
Average Pages per Session
↑ 136%
Compared to the previous 90 days
The average number of pages visited per session
Traffic by Agent Type
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI browsing activity by agent type over time
Top Visited Website Categories
Computers and Electronics
0.1%
Business and Industrial
0.0%
Arts and Entertainment
0.0%
Internet and Telecom
0.0%
Travel and Transportation
0.0%
Books and Literature
0.0%
Website categories with most activity
Top Agents
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
43.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
35.7%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
21.4%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
AGNT
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
0.0%
Agents doing the most AI browsing
Top Operators
Operators doing the most AI browsing
Robots.txt & Compliance
See which robots.txt rules are set across the web and how well agents follow them. An agent's Robots.txt Effectiveness measures the effectiveness of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked.
Effectiveness (Overall)
How often all agents across all agent types follow robots.txt rules
Top Rule-Following Agents
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
100.0%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
99.9%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
99.8%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
99.8%
INT
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
99.5%
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
99.4%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
99.2%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
99.2%
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
99.2%
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
99.1%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
99.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
99.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
99.0%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
98.9%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
98.8%
Agents with the best Robots.txt Effectiveness percentages
Top Rule-Breaking Agents
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
66.1%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
74.9%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
82.8%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
83.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
84.7%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
85.0%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
86.7%
UND
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
87.7%
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
88.6%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
89.1%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
91.9%
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
92.9%
FTCH
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
93.3%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
93.8%
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
94.4%
Agents with the worst Robots.txt Effectiveness percentages
Top Blocked Agents
Agents blocked by the most top websites
Spoofing & Security
See which agents are most frequently impersonated, and how spoofing activity changes over time. A visit is considered spoofed when it claims a recognized agent identity but fails that agent's supported authentication method, such as verified IP or HTTP message signature (Web Bot Auth).
Active Threat: AI Bot Spoofing Campaign
We are observing a widespread campaign impersonating AI bots to scan websites for vulnerabilities. The attacker appears to be targeting credential and configuration paths used by AI coding tools. Contact us for more information, or inspect your own traffic.
Spoofed Traffic by Agent Identity
The percentage of impersonated website traffic for each agent identity over time
Top Spoofed Agent Identities
The most impersonated agent identities
Recent Top Targeted Paths
/.config/anthropic/credentials/default.json
/.claude/settings.json
/.claude.json
/.hermes/.env
/.openclaw/.env
/.codex/config.toml
/.continue/config.json
/.aider.conf.yml
/service-account.json
/serviceaccountkey.json
/service_account.json
/firebase-adminsdk.json
/firebase-service-account.json
/.aws/credentials
/.aws/config
/.s3cfg
/.boto
/.npmrc
/.env.example
/.env.local
/.env.production
/.env.backup
/.env.old
/backend/.env
/api/.env
/admin/.env
/dockerfile
/docker-compose.yaml
/.docker/config.json
/terraform.tfstate
/credentials.json
/secrets.json
/secrets.yml
/key.json
/rclone.conf
Examples of recent top targeted request paths
AI Chat Referrals
See which AI platforms like ChatGPT, Perplexity, and Gemini cite websites and send them human referral traffic. Citations are estimated. Google's guide explains how websites can optimize their content to be more visible in AI chat responses (GEO).
Traffic by AI Chat Platform
ChatGPT
Claude
Copilot
DeepSeek
Gemini
Meta AI
Mistral
Perplexity
Referral activity by AI chat platform over time
Top Mentioned (Cited) Website Categories
Computers and Electronics
1.8%
Travel and Transportation
1.6%
Internet and Telecom
1.6%
Business and Industrial
1.4%
Website categories most frequently cited in AI chat responses
Top Clicked (Referred) Website Categories
Travel and Transportation
0.3%
Computers and Electronics
0.2%
Business and Industrial
0.1%
Internet and Telecom
0.0%
Website categories receiving the most referrals from AI chat
Methodology
Data Scope
The Index updates daily using completed days of traffic, security, and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. Percentage changes compare equal-length periods. Agent data comes from the Agent Directory, and website categories follow Google AdSense. Participating websites are not a random sample, and the qualifying set changes over time, so results show observed directional trends rather than a census of global web traffic.
Qualification & Aggregation
Only website-days meeting minimum activity and data-quality requirements are included; internal, test, incomplete, and anomalous data is excluded. Bot percentages use all visits, while AI chat referral percentages use estimated human visits after identified bots are removed. Rates are calculated per website-day and then averaged, giving each website equal weight regardless of traffic. Daily charts are not smoothed in order to preserve natural seasonality.
Measuring Robots.txt Effectiveness
Robots.txt Effectiveness estimates the traffic reduction associated with blocking an agent sitewide using Disallow: /. For each agent and day, the baseline allowed rate includes every qualifying website that allows it, even those with zero agent visits:
For each blocked website, expected visits are:
Given observed disallowed visits O, its score is:
A 0% score means visits met or exceeded expectations, 50% means half as many visits as expected, and 100% means none were observed.
Scores require meaningful activity across multiple allowed and blocked websites over at least 14 of the latest 28 completed days. Each qualifying website-day has equal weight, and the headline score gives each qualifying agent equal weight. Only verified visits count for agents supporting authentication. This observational score measures outcomes, not intent or causation.
Top Blocked Bots is calculated separately from robots.txt scans of Similarweb's top 1,000 websites.
Identifying Spoofed Bots
Spoofing statistics count visits that claim a known agent's identity but fail a supported authentication method, such as IP verification or HTTP message signatures (Web Bot Auth). Each agent's daily rate is calculated against total visits per website and then averaged across websites. A failure suggests impersonation but does not identify the actual software or operator. Agents without supported authentication are excluded.
AI Chat Citations & Referrals
AI chat referrals count observed human visits with a recognized AI platform in the referrer or campaign source; visits without that data cannot be attributed. Citation estimates use requests from agents that retrieve content for AI platforms. Those requests may inform a response but do not prove a user saw a citation, so citation results are directional rather than exact counts.
Frequently Asked Questions
Can journalists and media organizations use this data?
Absolutely. You may cite The Agentic Web Index with attribution and a link to this page. For interviews, fact-checking, background context, or a more specific breakdown for a story, contact us and include your deadline.
Do you work with researchers?
Absolutely. We welcome thoughtful research into how agents and bots are changing the web. Tell us about your research question, timeframe, and intended use. Depending on the scope and data constraints, we may be able to provide additional context, compare approaches, or explore a joint analysis.
Can I request a specific analysis?
Yes. If you need a breakdown by agent, operator, activity type, website category, or time period that is not shown here, contact us. When the underlying data supports it, we can examine the question and provide a focused analysis.
How do I see these trends on my own website?
Agent Analytics shows which agents and bots visit your website, what they access, and how their activity changes over time. AI Chat Referral Tracking measures the human traffic arriving from AI chat platforms. Automatic Robots.txt helps manage which bots can access your content.