AI crawlers and AI agents
Crawlers and fetchers run by AI companies: crawlers that collect training data, crawlers that build an AI search index, and agents that fetch a page when someone asks an assistant a question. Vireo keeps them out of your visitor numbers and counts the pages they took, which the AI assistants card sets against the readers those assistants sent back.
89 known bots from 57 operators. 86 have their own page with user agents, robots.txt tokens and sources. Vireo shows what these took next to the readers AI assistants sent back. See the AI assistants card.
| Bot | Operator | Type |
|---|---|---|
| Ai2Bot | Allen Institute for AI (Ai2) | AI Data Scraper |
| Ai2Bot-DeepResearchEval | The Allen Institute for Artificial Intelligence | AI Assistant |
| Ai2Bot-Dolma | The Allen Institute for Artificial Intelligence | AI Data Scraper |
| Amazonbot | Amazon | AI Search Crawler |
| Amazonbot-Video | Amazon.com, Inc. | Crawler |
| AmazonBuyForMe | Amazon.com, Inc. | AI Agent |
| Amzn-SearchBot | Amazon.com, Inc. | AI Search Crawler |
| Amzn-User | Amazon.com, Inc. | AI Assistant |
| Andibot | Andi | AI Search Crawler |
| Anthropic AI | Anthropic, PBC | AI Data Scraper |
| Applebot-Extended | Apple | AI Data Scraper |
| Bard-AI | Google Inc. | AI Search Crawler |
| Big Sur AI | Big Sur AI, Inc. | AI Assistant |
| Bytespider | ByteDance (Toutiao Search) | Search bot |
| ccBot crawler | Common Crawl | Crawler |
| Character-AI | Character Technologies, Inc. | AI Assistant |
| ChatGLM-Spider | Beijing Zhipu Huazhang Technology Co., Ltd. | AI Data Scraper |
| ChatGPT-Browser | OpenAI OpCo, LLC | AI Assistant |
| ChatGPT-User | OpenAI | AI Assistant |
| Claude-SearchBot | Anthropic | AI Search Crawler |
| Claude-User | Anthropic | AI Assistant |
| Claude-Web | Anthropic PBC | AI Search Crawler |
| ClaudeBot | Anthropic | AI Data Scraper |
| Cohere AI | Cohere, Inc. | AI Data Scraper |
| Cohere-Command | Cohere, Inc. | AI Data Scraper |
| cohere-training-data-crawler | Cohere, Inc. | AI Data Scraper |
| Crawlspace | Crawlspace Company | AI Data Scraper |
| DeepseekBot | Hangzhou Deep Search Artificiell Intelligens Basic Technology Research Co., Ltd. | AI Data Scraper |
| Devin | Cognition AI, Inc. | AI Assistant |
| Diffbot | Diffbot | AI Data Scraper |
| Diffbot-User | Diffbot | Listed by ai.robots.txt |
| Doximity-Diffbot | Diffbot Technologies Corp. | AI Data Scraper |
| FacebookBot | Meta Platforms, Inc. | Crawler |
| FirecrawlAgent | SideGuide Technologies, Inc. | AI Data Scraper |
| Gemini-AI | Google Inc. | AI Search Crawler |
| Gemini-Deep-Research | Google Inc. | AI Assistant |
| GitHubCopilotChat | Not known | AI Assistant |
| GoGuidesCrawler | GoGuides, LLC | AI Search Crawler |
| GoGuidesDiscoveryBot | GoGuides, LLC | AI Search Crawler |
| Google-CloudVertexBot | Google Inc. | AI Data Scraper |
| Google-Extended | AI Data Scraper | |
| Google-GeminiNotebook | Google Inc. | AI Assistant |
| Google-NotebookLM | Google Inc. | AI Assistant |
| Google-Pinpoint | Google Inc. | AI Assistant |
| GoogleAgent-Mariner | Google Inc. | AI Agent |
| GoogleAgent-Search | Google Inc. | AI Assistant |
| GPTBot | OpenAI | AI Data Scraper |
| Groq-Bot | Groq, Inc. | AI Data Scraper |
| HuggingFace-Bot | Hugging Face, Inc | AI Data Scraper |
| IAS Crawler | Integral Ad Science, Inc. | AI Data Scraper |
| IAS Wombles | Integral Ad Science, Inc. | AI Data Scraper |
| iAskBot | Ai Search Inc. | AI Data Scraper |
| IbouBot | Not known | AI Search Crawler |
| ImageSift | Castle Global, Inc. | AI Data Scraper |
| ImagesiftBot | ImageSift | Listed by ai.robots.txt |
| Kimi-SearchBot | Moonshot AI Pte. Ltd. | AI Search Crawler |
| Kimi-User | Moonshot AI Pte. Ltd. | AI Assistant |
| KimiBot | Moonshot AI Pte. Ltd. | AI Data Scraper |
| LinerBot | Liner | AI Assistant |
| Manus-User | Butterfly Effect Pte. Ltd. | AI Assistant |
| Meta-ExternalAgent | Meta | AI Assistant |
| MistralAI-Index | Mistral AI | AI Search Crawler |
| MistralAI-User | Mistral AI | AI Assistant |
| MistralAI-User/1.0 | Mistral AI | Listed by ai.robots.txt |
| NovaAct | Amazon.com, Inc. | AI Agent |
| OAI-SearchBot | OpenAI | AI Search Crawler |
| Omgilibot | Webz.io Ltd | Search bot |
| PanguBot | Huawei Technologies Co., Ltd. | AI Data Scraper |
| Perplexity-User | Perplexity | AI Assistant |
| PerplexityBot | Perplexity | AI Search Crawler |
| Poggio-Citations | Poggio Labs, Inc. | AI Assistant |
| ProRata | ProRataAI, Inc. | AI Search Crawler |
| QualifiedBot | Qualified.com, Inc. | AI Assistant |
| Replicate-Bot | Replicate, Inc. | AI Data Scraper |
| RunPod-Bot | RunPod Inc. | AI Data Scraper |
| SBIntuitionsBot | SB Intuitions Corp. | AI Data Scraper |
| ShapBot | Parallel Web Systems Inc. | AI Data Scraper |
| Sider | Sider AI | AI Data Scraper |
| TerraCotta | Ceramic, Inc. | AI Data Scraper |
| Timpibot | Timpi Inc. | Crawler |
| Together-Bot | Together Computer Inc. | AI Data Scraper |
| UdV-AI-Research | Not known | AI Data Scraper |
| webzio | Webz.io Ltd | AI Search Crawler |
| webzio-extended | Webz.io Ltd | AI Data Scraper |
| WRTNBot | Rutten Technologies Co., Ltd | AI Data Scraper |
| xAI-Bot | X.AI LLC | AI Data Scraper |
| YandexAdditionalBot | Yandex LLC | AI Search Crawler |
| YouBot | You.com | AI Search Crawler |
| ZanistaBot | Zanista AI LTD. | AI Search Crawler |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
Search engines
Crawlers that fetch pages to build a search index.
128 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots