Bot directory

993 known bots from 563 operators, grouped the way Vireo Analytics reports them. Look one up to see who runs it, what it does, its user agent and how to block it.

AI crawlers

AI companies run several kinds of bot: crawlers that gather training data, crawlers that build an AI search index, and fetchers that load a page because someone asked an assistant about it. Each one usually has its own robots.txt token, so you can allow one and block another. Vireo counts the pages these bots take and shows them next to the readers AI assistants send you. Here's how that looks.

Find a bot

336 bots have their own page, listed below. The rest appear on their category page.

BotOperatorVireo category
360 MonitoringPlesk International GmbHUptime monitors
360JK360 Security Technology Inc.Uptime monitors
360Spider360 SearchSearch engines
360Spider-Image360 SearchSearch engines
360Spider-Video360 SearchSearch engines
AboundexbotAboundex.comSearch engines
AcoonAcoon GmbHSearch engines
AddSearchBotAddSearch OyOther bots
aHrefs BotAhrefsSEO tools
AhrefsSiteAuditAhrefsUptime monitors
Ai2BotAllen Institute for AI (Ai2)AI crawlers
Ai2Bot-DeepResearchEvalThe Allen Institute for Artificial IntelligenceAI crawlers
Ai2Bot-DolmaThe Allen Institute for Artificial IntelligenceAI crawlers
aiHitBotaiHitOther bots
Alexa Site AuditAlexa InternetUptime monitors
Amazon ELBAWSUptime monitors
Amazon KendraAmazon.com, Inc.Other bots
AmazonbotAmazonAI crawlers
Amazonbot-VideoAmazon.com, Inc.AI crawlers
AmazonBuyForMeAmazon.com, Inc.AI crawlers
Amzn-SearchBotAmazon.com, Inc.AI crawlers
Amzn-UserAmazon.com, Inc.AI crawlers
AndibotAndiAI crawlers
Anthropic AIAnthropic, PBCAI crawlers
ApplebotAppleSearch engines
Applebot-ExtendedAppleAI crawlers
AppSignalBotAppSignalUptime monitors
archive.org botInternet Archive (Archive-It)Archivers and validators
atlassian-botAtlassianOther bots
AwarioTechFusion Ltd.Search engines
Baidu SpiderBaiduSearch engines
Bard-AIGoogle Inc.AI crawlers
BBC Forge URL MonitorBBCUptime monitors
BBC Page MonitorBBCUptime monitors
bedrockbotAmazonOther bots
Better Uptime BotBetter StackUptime monitors
Big Sur AIBig Sur AI, Inc.AI crawlers
BingBotMicrosoft (Bing)Search engines
Blackbox ExporterPrometheusUptime monitors
BlekkobotBlekkoSearch engines
BLEXBot CrawlerWebMeUpSEO tools
BlueskyBluesky Social PBCLink previews
BoardReaderEffyis IncSearch engines
Bountii BotBountii Inc.Search engines
BravebotBrave Software, Inc.Search engines
BrightBotBright Interactive LtdOther bots
Brightbot 1.0https://brightdata.com/brightbotOther bots
BuckHypefactors A/SUptime monitors
BuddyBotBuddyBotLearningOther bots
Butterfly RobotTopsy LabsSearch engines
BytespiderByteDance (Toutiao Search)AI crawlers
CatchpointCatchpoint SystemsUptime monitors
CATExploradorFundació puntCATSearch engines
ccBot crawlerCommon CrawlAI crawlers
Character-AICharacter Technologies, Inc.AI crawlers
ChartableChartableUptime monitors
ChatGLM-SpiderBeijing Zhipu Huazhang Technology Co., Ltd.AI crawlers
ChatGPT-BrowserOpenAI OpCo, LLCAI crawlers
ChatGPT-UserOpenAIAI crawlers
CheckHostCheckHostUptime monitors
Claude-SearchBotAnthropicAI crawlers
Claude-UserAnthropicAI crawlers
Claude-WebAnthropic PBCAI crawlers
ClaudeBotAnthropicAI crawlers
CloudFlare Always OnlineCloudflareUptime monitors
Cloudflare DiagnosticsCloudflareUptime monitors
Cloudflare ObservatoryCloudflareUptime monitors
Cloudflare Security InsightsCloudflareUptime monitors
Cloudflare Smart TransitCloudFlareUptime monitors
Cloudflare SSL DetectorCloudflareUptime monitors
Cloudflare Traffic ManagerCloudflareUptime monitors
CloudflareBrowserRenderingCrawlerCloudflare that returns rendered website content for research, monitoring, and AI data workflowsOther bots
Cốc Cốc BotCốc CốcSearch engines
Cohere AICohere, Inc.AI crawlers
Cohere-CommandCohere, Inc.AI crawlers
cohere-training-data-crawlerCohere, Inc.AI crawlers
CollectdCollectdUptime monitors
ContentKingConductorUptime monitors
CotoyogiJoint Support-Center for Data Science Research (ROIS-DS)Other bots
CragCrawlerCragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping servicesOther bots
CrawlspaceCrawlspace CompanyAI crawlers
CSS Certificate SpiderCertified Security SolutionsOther bots
Datadog AgentDatadogUptime monitors
Datenbank CrawlerDatenbankOther bots
DaumDaumSearch engines
DeepseekBotHangzhou Deep Search Artificiell Intelligens Basic Technology Research Co., Ltd.AI crawlers
DevinCognition AI, Inc.AI crawlers
DiffbotDiffbotAI crawlers
Diffbot-UserDiffbotAI crawlers
DiscobotDiscovery EngineSearch engines
DotBotMozSEO tools
DoubaoBotByteDanceOther bots
Doximity-DiffbotDiffbot Technologies Corp.AI crawlers
DuckAssistBotDuckDuckGoSearch engines
DuckDuckBotDuckDuckGoSearch engines
DynatraceSyntheticDynatraceUptime monitors
Easou Spidereasou ICPSearch engines
Echobot BotEchoboxOther bots
EchoboxBotEchoboxOther bots
ERNIEBotBaiduOther bots
ExaBotDassault SystèmesSearch engines
ExactSeek CrawlerJayde Online, Inc.Search engines
ExaSearchBotExaOther bots
EyeMonitEyeMonitUptime monitors
Facebook CrawlerMetaLink previews
FacebookBotMeta Platforms, Inc.AI crawlers
facebookexternalhitMeta/FacebookLink previews
Factset_spyderbotFactsetOther bots
FirecrawlAgentSideGuide Technologies, Inc.AI crawlers
FreeWebMonitoringGreenWave OnlineUptime monitors
GeistHaus-PageFetcherGeistHaus, a company developing AI systems for therapy and psychological assessmentOther bots
Gemini-AIGoogle Inc.AI crawlers
Gemini-Deep-ResearchGoogle Inc.AI crawlers
Ghost InspectorGhost InspectorUptime monitors
GigablastMatt WellsSearch engines
GoGuidesCrawlerGoGuides, LLCAI crawlers
GoGuidesDiscoveryBotGoGuides, LLCAI crawlers
GooNTT ResonantSearch engines
Google PageSpeed InsightsGoogleUptime monitors
Google Stackdriver MonitoringGoogle CloudUptime monitors
Google StoreBotGoogleSearch engines
Google Transparency ReportGoogle Inc.Uptime monitors
Google-CloudVertexBotGoogle Inc.AI crawlers
Google-ExtendedGoogleAI crawlers
Google-FirebaseGoogle Inc.Other bots
Google-GeminiNotebookGoogle Inc.AI crawlers
Google-NotebookLMGoogle Inc.AI crawlers
Google-PinpointGoogleAI crawlers
Google-Read-AloudGoogleOther bots
Google-SafetyGoogleOther bots
GoogleAgent-MarinerGoogle Inc.AI crawlers
GoogleAgent-SearchGoogle Inc.AI crawlers
GooglebotGoogleSearch engines
Googlebot NewsGoogleSearch engines
GoogleOtherGoogleSearch engines
GoogleOther-ImageGoogleSearch engines
GoogleOther-VideoGoogleSearch engines
GPTBotOpenAIAI crawlers
GrafanaGrafana LabsUptime monitors
Groq-BotGroq, Inc.AI crawlers
HTTPMontowards GmbHUptime monitors
HuggingFace-BotHugging Face, IncAI crawlers
IAS CrawlerIntegral Ad ScienceAI crawlers
IAS WomblesIntegral Ad ScienceAI crawlers
iAskBotAi Search Inc.AI crawlers
iaskspider/2.0iAskOther bots
ICC-CrawlerNICTOther bots
ImageSiftCastle Global, Inc.AI crawlers
ImagesiftBotImageSiftAI crawlers
img2datasetimg2datasetOther bots
ISSCyberRiskCrawlerISS-CorporateOther bots
kagi-fetcherKagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…Other bots
KagibotKagiSearch engines
KeenableBotKeenableOther bots
KeybaseBotZoom Video Communications, IncLink previews
Kimi-SearchBotMoonshot AI Pte. Ltd.AI crawlers
Kimi-UserMoonshot AI Pte. Ltd.AI crawlers
KimiBotMoonshot AI Pte. Ltd.AI crawlers
KlaviyoAIBotKlaviyoOther bots
LCCUniversität LeipzigSearch engines
LinerBotLinerAI crawlers
Linguee BotLingueeOther bots
Linkdex BotMojeek Ltd.Search engines
LinkedIn BotLinkedInLink previews
LinkpadBotSolomono LLCSEO tools
LumarLumarSearch engines
Magpie-CrawlerBrandwatchLink previews
Mail.Ru BotMail.Ru GroupSearch engines
Manus-UserButterfly Effect Pte. Ltd.AI crawlers
Meanpath BotMeanpathSearch engines
Meta-ExternalAdsMetaLink previews
Meta-ExternalAgentMetaAI crawlers
Meta-ExternalFetcherMetaLink previews
Meta-WebIndexerMeta Platforms, Inc.Link previews
MistralAI-IndexMistral AIAI crawlers
MistralAI-UserMistral AIAI crawlers
MistralAI-User/1.0Mistral AIAI crawlers
MJ12 BotMajesticSEO tools
MnogosearchLavtech.Com Corp.Search engines
MojeekBotMojeekSearch engines
Monitor.UsMonitor.UsUptime monitors
Montastic MonitorMetadot, Corp.Uptime monitors
MuninMuninUptime monitors
NagetBotNaget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts)Other bots
Nagios check_httpNagios Plugins Development TeamUptime monitors
NeevabotNeeva Inc.SEO tools
netEstate Imprint CrawlernetEstateOther bots
NodePingNodePingUptime monitors
NovaActAmazon.com, Inc.AI crawlers
OAI-AdsBotOpenAIOther bots
OAI-SearchBotOpenAIAI crawlers
oBotIBM Germany Research & Development GmbHSearch engines
Oh DearOh DearUptime monitors
OmgilibotWebz.io LtdAI crawlers
OnlineOrNot BotOnlineOrNotUptime monitors
Openindex SpiderOpenindexSearch engines
Orange BotOrangeSearch engines
Page Modified PingerValley Hosting, LLCUptime monitors
PanguBotHuawei Technologies Co., Ltd.AI crawlers
PanscientPanscient, Inc.Other bots
PaperLiBotSmallrivers SASearch engines
Perplexity-UserPerplexityAI crawlers
PerplexityBotPerplexityAI crawlers
Petal BotHuawei (Petal Search)Search engines
PhindBotphindOther bots
PHP Server MonitorPHP Server MonitorUptime monitors
Picsearch botPicsearchSearch engines
Pingdom BotSolarWinds (Pingdom)Uptime monitors
PinterestPinterestLink previews
Poggio-CitationsPoggio Labs, Inc.AI crawlers
Poseidon Research CrawlerPoseidon ResearchOther bots
PresearchPresearch.com Inc.Search engines
ProRataProRataAI crawlers
QualifiedBotQualified.com, Inc.AI crawlers
Querit-SearchBotQuerit that indexes web content for their search API service, which is designed to provide real-time search results for larg…Other bots
QueritBotQuerit, a company providing a search API for large language model integrationOther bots
QuillBotQuillbotOther bots
quillbot.comQuillbotOther bots
QwantbotQwantOther bots
QwenBotAlibabaOther bots
Reddit Botreddit inc.Link previews
ReflectionbotReflectionOther bots
Replicate-BotReplicate, Inc.AI crawlers
RocketMonitorBotRadio Mast, Inc.Uptime monitors
RogerbotMozSEO tools
RunPod-BotRunPod Inc.AI crawlers
RuxitSyntheticDynatraceUptime monitors
SBIntuitionsBotSB Intuitions Corp.AI crawlers
ScrapyZyteScripts and clients
Screaming Frog SEO SpiderScreaming Frog LtdSEO tools
SemrushBotSemrushSEO tools
SemrushBot-OCOBSemrushSEO tools
SemrushBot-SWASemrushSEO tools
SentiBotSentiOne S.A.Search engines
serpstatbotSerpstatSEO tools
Seznam BotSeznam.czSearch engines
ShapBotParallel Web Systems Inc.AI crawlers
ShopWikiShopWiki Corp.Search engines
SiderSider AIAI crawlers
Sidetrade indexer botSidetradeOther bots
Site24x7 Defacement MonitorSite24x7Uptime monitors
Site24x7 Website MonitoringSite24x7Uptime monitors
SiteAuditBotSemrushSEO tools
SiteimproveSiteimproveSearch engines
Sixy.chManuel KasperUptime monitors
SlackbotSlackLink previews
SMTBotSimilarTech Ltd.Search engines
Sogou SpiderSogouSearch engines
Soso SpiderTencent HoldingsSearch engines
StatusCakeStatusCakeUptime monitors
Survey BotDomain ToolsSearch engines
TerraCottaCeramic, Inc.AI crawlers
ThinkbotThinkbotOther bots
ThousandEyesCisco Systems, Inc.Uptime monitors
TikTokSpiderTikTok Inc.Search engines
TimpibotTimpi Inc.AI crawlers
TinEye CrawlerIdée Inc.Search engines
Together-BotTogether Computer Inc.AI crawlers
TongyiBotAlibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answersOther bots
TraceMyFileIdee Inc.Search engines
TwitterbotTwitterLink previews
UptimebotUptime.comUptime monitors
UptimeRobotUptimeRobotUptime monitors
UptimiaUptimiaUptime monitors
URLinspectorLinkResearchTools GmbHUptime monitors
Velen Public Web CrawlerVelen CrawlerOther bots
WARDBotW.A.R.D. LLCUptime monitors
WeboramaWeborama SASearch engines
WebSitePulseWebSitePulseUptime monitors
webzioWebz.ioAI crawlers
webzio-extendedWebz.io LtdAI crawlers
WeSEE:SearchWeSEE LtdSearch engines
WikiDoFotolitografie Fiorentine di Becchi Antonio s.n.c.Search engines
WooRankWooRank sprlSearch engines
WotboxWotboxSearch engines
wpbotQuantumCloudOther bots
WRTNBotRutten Technologies Co., LtdAI crawlers
xAI-BotX.AI LLCAI crawlers
YaCyYaCySearch engines
YaDirectFetcherYandex LLCSearch engines
Yahoo GeminiYahoo! Inc.Uptime monitors
Yahoo! SlurpYahooSearch engines
YaKLinkfluence SASOther bots
Yandex.Gazeta BotYandex LLCSearch engines
YandexAccessibilityBotYandexSearch engines
YandexAdditionalBotYandexAI crawlers
YandexAdNetYandexSearch engines
YandexAntivirusYandex LLCSearch engines
YandexBlogsYandexSearch engines
YandexBotYandexSearch engines
YandexBot-MirrorDetectorYandexSearch engines
YandexCalendarYandexSearch engines
YandexComBotYandexSearch engines
YandexDialogsYandexSearch engines
YandexDirectYandexSearch engines
YandexDirectDynYandexSearch engines
YandexFaviconsYandexSearch engines
YandexForDomainYandex LLCSearch engines
YandexImageResizerYandex LLCSearch engines
YandexImagesYandexSearch engines
YandexMarketYandexSearch engines
YandexMediaYandexSearch engines
YandexMedianaBotYandex LLCSearch engines
YandexMetrikaYandexSearch engines
YandexMobileBotYandexSearch engines
YandexMobileScreenShotBotYandexSearch engines
YandexNewsYandex LLCSearch engines
YandexNewslinksYandex LLCSearch engines
YandexOntoDBYandexSearch engines
YandexOntoDBAPIYandexSearch engines
YandexPagecheckerYandexSearch engines
YandexPartnerYandexSearch engines
YandexRCAYandexSearch engines
YandexRenderResourcesBotYandexSearch engines
YandexScreenshotBotYandexSearch engines
YandexSearchShopYandexSearch engines
YandexSitelinksYandexSearch engines
YandexSpravBotYandexSearch engines
YandexTrackerYandexSearch engines
YandexTurboYandex LLCSearch engines
YandexUserproxyYandexSearch engines
YandexVerticalsYandexSearch engines
YandexVertisYandexSearch engines
YandexVideoYandexSearch engines
YandexVideoParserYandexSearch engines
YandexWebmasterYandexSearch engines
Yeti/NaverbotNaverSearch engines
YioopBotChris PollettSearch engines
YiyanBotBaidu that fetches web content for the yiyanOther bots
yoozBotyoozSearch engines
Yottaa Site MonitorYottaaUptime monitors
YouBotYou.comAI crawlers
Youdao BotNetEase, Inc.Search engines
Yunyun BotYunYunSearch engines
ZanistaBotZanista AI LTD.AI crawlers
ZumBotZUM internetSearch engines

About this data

Bot names, operators, types, user agents and patterns come from Matomo Device Detector, released under the GNU LGPL v3. Vireo Analytics' own bot signatures are built from the same file. Details on AI crawlers, including their robots.txt tokens, come from the ai.robots.txt project (MIT license). Where an operator documents its crawler, the page quotes that documentation and links to it; those pages were checked on 2026-10-06.

Each bot's category is what Vireo Analytics' bot filter actually returns for that bot's user agent, run against the plugin's code. Bots the filter doesn't recognise aren't listed. Facts that no source states, such as whether an obscure crawler runs JavaScript, are left out rather than guessed.

Device Detector commit e83ce7a, ai.robots.txt commit 9ad8a47.