Bot directory
993 known bots from 563 operators, grouped the way Vireo Analytics reports them. Look one up to see who runs it, what it does, its user agent and how to block it.
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
Search engines
Crawlers that fetch pages to build a search index.
128 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots
AI crawlers
AI companies run several kinds of bot: crawlers that gather training data, crawlers that build an AI search index, and fetchers that load a page because someone asked an assistant about it. Each one usually has its own robots.txt token, so you can allow one and block another. Vireo counts the pages these bots take and shows them next to the readers AI assistants send you. Here's how that looks.
- Ai2Bot
- Ai2Bot-DeepResearchEval
- Ai2Bot-Dolma
- Amazonbot
- Amazonbot-Video
- AmazonBuyForMe
- Amzn-SearchBot
- Amzn-User
- Andibot
- Anthropic AI
- Applebot-Extended
- Bard-AI
- Big Sur AI
- Bytespider
- ccBot crawler
- Character-AI
- ChatGLM-Spider
- ChatGPT-Browser
- ChatGPT-User
- Claude-SearchBot
- Claude-User
- Claude-Web
- ClaudeBot
- Cohere AI
- Cohere-Command
- cohere-training-data-crawler
- Crawlspace
- DeepseekBot
- Devin
- Diffbot
- Diffbot-User
- Doximity-Diffbot
- FacebookBot
- FirecrawlAgent
- Gemini-AI
- Gemini-Deep-Research
- GoGuidesCrawler
- GoGuidesDiscoveryBot
- Google-CloudVertexBot
- Google-Extended
- Google-GeminiNotebook
- Google-NotebookLM
- Google-Pinpoint
- GoogleAgent-Mariner
- GoogleAgent-Search
- GPTBot
- Groq-Bot
- HuggingFace-Bot
- IAS Crawler
- IAS Wombles
- iAskBot
- ImageSift
- ImagesiftBot
- Kimi-SearchBot
- Kimi-User
- KimiBot
- LinerBot
- Manus-User
- Meta-ExternalAgent
- MistralAI-Index
- MistralAI-User
- MistralAI-User/1.0
- NovaAct
- OAI-SearchBot
- Omgilibot
- PanguBot
- Perplexity-User
- PerplexityBot
- Poggio-Citations
- ProRata
- QualifiedBot
- Replicate-Bot
- RunPod-Bot
- SBIntuitionsBot
- ShapBot
- Sider
- TerraCotta
- Timpibot
- Together-Bot
- webzio
- webzio-extended
- WRTNBot
- xAI-Bot
- YandexAdditionalBot
- YouBot
- ZanistaBot
Find a bot
336 bots have their own page, listed below. The rest appear on their category page.
About this data
Bot names, operators, types, user agents and patterns come from Matomo Device Detector, released under the GNU LGPL v3. Vireo Analytics' own bot signatures are built from the same file. Details on AI crawlers, including their robots.txt tokens, come from the ai.robots.txt project (MIT license). Where an operator documents its crawler, the page quotes that documentation and links to it; those pages were checked on 2026-10-06.
Each bot's category is what Vireo Analytics' bot filter actually returns for that bot's user agent, run against the plugin's code. Bots the filter doesn't recognise aren't listed. Facts that no source states, such as whether an obscure crawler runs JavaScript, are left out rather than guessed.
Device Detector commit e83ce7a, ai.robots.txt commit 9ad8a47.