SEO tool crawlers
Crawlers run by SEO and backlink tools to map links between sites and audit pages. They aren't readers, and some of them crawl often.
17 known bots from 10 operators. 13 have their own page with user agents, robots.txt tokens and sources.
| Bot | Operator | Type |
|---|---|---|
| aHrefs Bot | Ahrefs | Crawler |
| Barkrowler | Not known | Crawler |
| BLEXBot Crawler | WebMeUp | Crawler |
| DataForSeoBot | Not known | Crawler |
| DotBot | Moz | Crawler |
| LinkpadBot | Solomono LLC | Crawler |
| MegaIndex | Not known | Crawler |
| MJ12 Bot | Majestic | Search bot |
| Neevabot | Neeva Inc. | Search bot |
| Rogerbot | Moz | Crawler |
| Screaming Frog SEO Spider | Screaming Frog Ltd | Crawler |
| SemrushBot | Semrush | Crawler |
| SemrushBot-OCOB | Semrush | Listed by ai.robots.txt |
| SemrushBot-SWA | Semrush | Listed by ai.robots.txt |
| serpstatbot | Netpeak Ltd | Crawler |
| SiteAuditBot | Semrush Inc. | Crawler |
| ZoominfoBot | Not known | Crawler |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
Search engines
Crawlers that fetch pages to build a search index.
128 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots