Web archives and validators
Web archives fetch a page to save a copy, and validators fetch it to check its markup or certificates.
16 known bots from 8 operators. 1 have their own page with user agents, robots.txt tokens and sources.
| Bot | Operator | Type |
|---|---|---|
| Alexa Crawler | Alexa Internet | Search bot |
| archive.org bot | Internet Archive (Archive-It) | Crawler |
| CSSCheck | Not known | Validator |
| eZ Publish Link Validator | eZ Systems AS | Crawler |
| Google Structured Data Testing Tool | Google Inc. | Validator |
| Google-Site-Verification | Google Inc. | Validator |
| MoodleBot Linkchecker | Moodle Pty Ltd | Search bot |
| SSL Labs | SSL Labs | Validator |
| Startpagina Linkchecker | Startpagina B.V. | Search bot |
| W3C CSS Validator | W3C | Validator |
| W3C I18N Checker | W3C | Validator |
| W3C Markup Validation Service | W3C | Validator |
| W3C MobileOK Checker | W3C | Validator |
| W3C P3P Validator | W3C | Validator |
| W3C Unified Validator | W3C | Validator |
| WDG HTML Validator | Not known | Validator |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
Search engines
Crawlers that fetch pages to build a search index.
128 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots