Scripts and HTTP clients
HTTP libraries and command-line clients show up when someone requests your pages from code rather than a browser.
5 known bots from 5 operators. 1 have their own page with user agents, robots.txt tokens and sources.
| Bot | Operator | Type |
|---|---|---|
| IsItWP | WPBeginner, LLC | Crawler |
| Scrapy | Zyte | Crawler |
| svEye | Semantic Visions, s.r.o. | Crawler |
| W3C Link Checker | W3C | Validator |
| WordPress | Wordpress.org | Service Agent |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
Search engines
Crawlers that fetch pages to build a search index.
128 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Link previews
Fetchers that build a preview card when someone shares your link.
20 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots