Search engine crawlers
Search engine crawlers fetch your pages to build their index. They're a big part of why server logs show more requests than an analytics report.
128 known bots from 67 operators. 113 have their own page with user agents, robots.txt tokens and sources.
| Bot | Operator | Type |
|---|---|---|
| 360Spider | Online Media Group, Inc. | Search bot |
| 360Spider-Image | Online Media Group, Inc. | Search bot |
| 360Spider-Video | Online Media Group, Inc. | Search bot |
| Aboundexbot | Aboundex.com | Search bot |
| Acoon | Acoon GmbH | Search bot |
| Applebot | Apple | AI Search Crawler |
| Ask Jeeves | Ask Jeeves Inc. | Search bot |
| Awario | TechFusion Ltd. | Feed Fetcher |
| Baidu Spider | Baidu | Search bot |
| BingBot | Microsoft (Bing) | Search bot |
| Blekkobot | Blekko | Search bot |
| BoardReader | Effyis Inc | Search bot |
| Bountii Bot | Bountii Inc. | Search bot |
| Bravebot | Brave Software, Inc. | Search bot |
| Butterfly Robot | Topsy Labs | Search bot |
| CATExplorador | Fundació puntCAT | Search bot |
| Centurybot | Not known | Search bot |
| cl0q-crawler | Not known | Search bot |
| Cốc Cốc Bot | Cốc Cốc | Search bot |
| Daum | Daum Communications Corp. | Search bot |
| Dazoobot | DAZOO.FR | Search bot |
| Discobot | Discovery Engine | Search bot |
| DuckAssistBot | DuckDuckGo | AI Assistant |
| DuckDuckBot | DuckDuckGo | Search bot |
| Easou Spider | easou ICP | Search bot |
| ExaBot | Dassault Systèmes | Crawler |
| ExactSeek Crawler | Jayde Online, Inc. | Search bot |
| FindiseBot | Not known | Search bot |
| Gigablast | Matt Wells | Search bot |
| Goo | NTT Resonant | Search bot |
| Google StoreBot | Crawler | |
| google-xrawler | Google Inc. | Search bot |
| Googlebot | Search bot | |
| Googlebot News | Google Inc. | Search bot |
| GoogleOther | Listed by ai.robots.txt | |
| GoogleOther-Image | Listed by ai.robots.txt | |
| GoogleOther-Video | Listed by ai.robots.txt | |
| Heexybot | Not known | Search bot |
| Inktomi Slurp | Not known | |
| Kagibot | Kagi Inc. | Search bot |
| LCC | Universität Leipzig | Search bot |
| Linkdex Bot | Mojeek Ltd. | Search bot |
| Lumar | Lumar | Crawler |
| Mail.Ru Bot | Mail.Ru Group | Search bot |
| Meanpath Bot | Meanpath | Search bot |
| Mnogosearch | Lavtech.Com Corp. | Search bot |
| MojeekBot | Mojeek Ltd. | Search bot |
| Netcraft Survey Bot | Netcraft | Search bot |
| oBot | IBM Germany Research & Development GmbH | Search bot |
| Openindex Spider | Openindex B.V. | Search bot |
| Orange Bot | Orange | Search bot |
| OrlightBot | Not known | Search bot |
| PaperLiBot | Smallrivers SA | Search bot |
| Petal Bot | Huawei (Petal Search) | Crawler |
| Picsearch bot | Picsearch | Search bot |
| Presearch | Presearch.com Inc. | Search bot |
| PriEcoBot | Not known | Search bot |
| SentiBot | SentiOne S.A. | AI Data Scraper |
| Seznam Bot | Seznam.cz | Search bot |
| ShopWiki | ShopWiki Corp. | Search tools |
| Siteimprove | Siteimprove GmbH | Search bot |
| SMTBot | SimilarTech Ltd. | Search bot |
| Sogou Spider | Sohu, Inc. | Search bot |
| Soso Spider | Tencent Holdings | Search bot |
| Survey Bot | Domain Tools | Search bot |
| Tarmot Gezgin | Not known | Search bot |
| TikTokSpider | TikTok Inc. | Search bot |
| TinEye Crawler | Idée Inc. | Search bot |
| TraceMyFile | Idee Inc. | Search bot |
| urlsuma | Not known | Search bot |
| URLSuMaBot | Not known | Crawler |
| vuhuvBot | Not known | Search bot |
| Weborama | Weborama SA | Search bot |
| WeSEE:Search | WeSEE Ltd | Search bot |
| WikiDo | Fotolitografie Fiorentine di Becchi Antonio s.n.c. | Search bot |
| WooRank | WooRank sprl | Search bot |
| Wotbox | Wotbox | Search bot |
| YaCy | YaCy | Search bot |
| YaDirectFetcher | Yandex LLC | Search bot |
| Yahoo! Slurp | Yahoo! Inc. | Search bot |
| Yandex.Gazeta Bot | Yandex LLC | Search bot |
| YandexAccessibilityBot | Yandex LLC | Search bot |
| YandexAdNet | Yandex LLC | Search bot |
| YandexAntivirus | Yandex LLC | Search bot |
| YandexBlogs | Yandex LLC | Search bot |
| YandexBot | Yandex | Search bot |
| YandexBot-MirrorDetector | Yandex LLC | Crawler |
| YandexCalendar | Yandex LLC | Search bot |
| YandexComBot | Yandex LLC | Search bot |
| YandexDialogs | Yandex LLC | Search bot |
| YandexDirect | Yandex LLC | Search bot |
| YandexDirectDyn | Yandex LLC | Search bot |
| YandexFavicons | Yandex LLC | Search bot |
| YandexForDomain | Yandex LLC | Search bot |
| YandexImageResizer | Yandex LLC | Search bot |
| YandexImages | Yandex LLC | Search bot |
| YandexMarket | Yandex LLC | Search bot |
| YandexMedia | Yandex LLC | Search bot |
| YandexMedianaBot | Yandex LLC | Search bot |
| YandexMetrika | Yandex LLC | Search bot |
| YandexMobileBot | Yandex LLC | Search bot |
| YandexMobileScreenShotBot | Yandex LLC | Search bot |
| YandexNews | Yandex LLC | Search bot |
| YandexNewslinks | Yandex LLC | Search bot |
| YandexOntoDB | Yandex LLC | Search bot |
| YandexOntoDBAPI | Yandex LLC | Search bot |
| YandexPagechecker | Yandex LLC | Search bot |
| YandexPartner | Yandex LLC | Search bot |
| YandexRCA | Yandex LLC | Search bot |
| YandexRenderResourcesBot | Yandex LLC | Search bot |
| YandexScreenshotBot | Yandex LLC | Search bot |
| YandexSearchShop | Yandex LLC | Search bot |
| YandexSitelinks | Yandex LLC | Search bot |
| YandexSpravBot | Yandex LLC | Search bot |
| YandexTracker | Yandex LLC | Search bot |
| YandexTurbo | Yandex LLC | Search bot |
| YandexUserproxy | Yandex LLC | Search bot |
| YandexVerticals | Yandex LLC | Search bot |
| YandexVertis | Yandex LLC | Search bot |
| YandexVideo | Yandex LLC | Search bot |
| YandexVideoParser | Yandex LLC | Search bot |
| YandexWebmaster | Yandex LLC | Search bot |
| Yeti/Naverbot | Naver | Search bot |
| YioopBot | Chris Pollett | Search bot |
| yoozBot | yooz | Search bot |
| Youdao Bot | NetEase, Inc. | Search bot |
| Yunyun Bot | YunYun | Search bot |
| ZumBot | ZUM internet | Search bot |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots