Search engine crawlers

Search engine crawlers fetch your pages to build their index. They're a big part of why server logs show more requests than an analytics report.

128 known bots from 68 operators. 113 have their own page with user agents, robots.txt tokens and sources.

BotOperatorType
360Spider360 SearchSearch bot
360Spider-Image360 SearchSearch bot
360Spider-Video360 SearchSearch bot
AboundexbotAboundex.comSearch bot
AcoonAcoon GmbHSearch bot
ApplebotAppleAI Search Crawler
Ask JeevesAsk Jeeves Inc.Search bot
AwarioTechFusion Ltd.Feed Fetcher
Baidu SpiderBaiduSearch bot
BingBotMicrosoft (Bing)Search bot
BlekkobotBlekkoSearch bot
BoardReaderEffyis IncSearch bot
Bountii BotBountii Inc.Search bot
BravebotBrave Software, Inc.Search bot
Butterfly RobotTopsy LabsSearch bot
CATExploradorFundació puntCATSearch bot
CenturybotNot knownSearch bot
cl0q-crawlerNot knownSearch bot
Cốc Cốc BotCốc CốcSearch bot
DaumDaumSearch bot
DazoobotDAZOO.FRSearch bot
DiscobotDiscovery EngineSearch bot
DuckAssistBotDuckDuckGoAI Assistant
DuckDuckBotDuckDuckGoSearch bot
Easou Spidereasou ICPSearch bot
ExaBotDassault SystèmesCrawler
ExactSeek CrawlerJayde Online, Inc.Search bot
FindiseBotNot knownSearch bot
GigablastMatt WellsSearch bot
GooNTT ResonantSearch bot
Google StoreBotGoogleCrawler
google-xrawlerGoogle Inc.Search bot
GooglebotGoogleSearch bot
Googlebot NewsGoogleSearch bot
GoogleOtherGoogleListed by ai.robots.txt
GoogleOther-ImageGoogleListed by ai.robots.txt
GoogleOther-VideoGoogleListed by ai.robots.txt
HeexybotNot knownSearch bot
Inktomi SlurpNot known
KagibotKagiSearch bot
LCCUniversität LeipzigSearch bot
Linkdex BotMojeek Ltd.Search bot
LumarLumarCrawler
Mail.Ru BotMail.Ru GroupSearch bot
Meanpath BotMeanpathSearch bot
MnogosearchLavtech.Com Corp.Search bot
MojeekBotMojeekSearch bot
Netcraft Survey BotNetcraftSearch bot
oBotIBM Germany Research & Development GmbHSearch bot
Openindex SpiderOpenindexSearch bot
Orange BotOrangeSearch bot
OrlightBotNot knownSearch bot
PaperLiBotSmallrivers SASearch bot
Petal BotHuawei (Petal Search)Crawler
Picsearch botPicsearchSearch bot
PresearchPresearch.com Inc.Search bot
PriEcoBotNot knownSearch bot
SentiBotSentiOne S.A.AI Data Scraper
Seznam BotSeznam.czSearch bot
ShopWikiShopWiki Corp.Search tools
SiteimproveSiteimproveSearch bot
SMTBotSimilarTech Ltd.Search bot
Sogou SpiderSogouSearch bot
Soso SpiderTencent HoldingsSearch bot
Survey BotDomain ToolsSearch bot
Tarmot GezginNot knownSearch bot
TikTokSpiderTikTok Inc.Search bot
TinEye CrawlerIdée Inc.Search bot
TraceMyFileIdee Inc.Search bot
urlsumaNot knownSearch bot
URLSuMaBotNot knownCrawler
vuhuvBotNot knownSearch bot
WeboramaWeborama SASearch bot
WeSEE:SearchWeSEE LtdSearch bot
WikiDoFotolitografie Fiorentine di Becchi Antonio s.n.c.Search bot
WooRankWooRank sprlSearch bot
WotboxWotboxSearch bot
YaCyYaCySearch bot
YaDirectFetcherYandex LLCSearch bot
Yahoo! SlurpYahooSearch bot
Yandex.Gazeta BotYandex LLCSearch bot
YandexAccessibilityBotYandexSearch bot
YandexAdNetYandexSearch bot
YandexAntivirusYandex LLCSearch bot
YandexBlogsYandexSearch bot
YandexBotYandexSearch bot
YandexBot-MirrorDetectorYandexCrawler
YandexCalendarYandexSearch bot
YandexComBotYandexSearch bot
YandexDialogsYandexSearch bot
YandexDirectYandexSearch bot
YandexDirectDynYandexSearch bot
YandexFaviconsYandexSearch bot
YandexForDomainYandex LLCSearch bot
YandexImageResizerYandex LLCSearch bot
YandexImagesYandexSearch bot
YandexMarketYandexSearch bot
YandexMediaYandexSearch bot
YandexMedianaBotYandex LLCSearch bot
YandexMetrikaYandexSearch bot
YandexMobileBotYandexSearch bot
YandexMobileScreenShotBotYandexSearch bot
YandexNewsYandex LLCSearch bot
YandexNewslinksYandex LLCSearch bot
YandexOntoDBYandexSearch bot
YandexOntoDBAPIYandexSearch bot
YandexPagecheckerYandexSearch bot
YandexPartnerYandexSearch bot
YandexRCAYandexSearch bot
YandexRenderResourcesBotYandexSearch bot
YandexScreenshotBotYandexSearch bot
YandexSearchShopYandexSearch bot
YandexSitelinksYandexSearch bot
YandexSpravBotYandexSearch bot
YandexTrackerYandexSearch bot
YandexTurboYandex LLCSearch bot
YandexUserproxyYandexSearch bot
YandexVerticalsYandexSearch bot
YandexVertisYandexSearch bot
YandexVideoYandexSearch bot
YandexVideoParserYandexSearch bot
YandexWebmasterYandexSearch bot
Yeti/NaverbotNaverSearch bot
YioopBotChris PollettSearch bot
yoozBotyoozSearch bot
Youdao BotNetEase, Inc.Search bot
Yunyun BotYunYunSearch bot
ZumBotZUM internetSearch bot

Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.

Not sure what a request in your logs is? Paste its user agent into the bot checker.

Other categories