Link preview fetchers
When someone shares your URL on a social network or in a chat app, the service fetches the page to build a preview card. That fetch is a bot, not the person who shared it.
20 known bots from 13 operators. 13 have their own page with user agents, robots.txt tokens and sources.
| Bot | Operator | Type |
|---|---|---|
| AddThis.com | Clearspring Technologies, Inc. | Social Media Agent |
| Bluesky | Bluesky Social PBC | Social Media Agent |
| Discord Bot | Not known | Service Agent |
| Facebook Crawler | Meta | Social Media Agent |
| facebookexternalhit | Meta/Facebook | Listed by ai.robots.txt |
| FediList | Not known | Social Media Agent |
| Friendica | Not known | Social Media Agent |
| KeybaseBot | Zoom Video Communications, Inc | Social Media Agent |
| LinkedIn Bot | Social Media Agent | |
| Magpie-Crawler | Brandwatch | Social Media Agent |
| Mastodon Bot | Not known | Social Media Agent |
| Meta-ExternalAds | Meta Platforms, Inc. | Social Media Agent |
| Meta-ExternalFetcher | Meta | Social Media Agent |
| Meta-WebIndexer | Meta Platforms, Inc. | Social Media Agent |
| Crawler | ||
| Reddit Bot | reddit inc. | Social Media Agent |
| Skype URI Preview | Skype Communications S.à.r.l. | Service Agent |
| Slackbot | Slack | Crawler |
| TelegramBot | Not known | |
| Twitterbot | Social Media Agent |
Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.
Other categories
AI crawlers
Training crawlers, AI search indexes and assistants fetching a page for a user.
89 bots
Search engines
Crawlers that fetch pages to build a search index.
128 bots
SEO tools
Backlink and site audit crawlers run by SEO tools.
17 bots
Uptime monitors
Services that request a page on a schedule to check it responds.
69 bots
Headless browsers
Lighthouse, GTmetrix and other tools that load pages like a browser.
8 bots
Scripts and clients
HTTP libraries and command-line clients making requests from code.
5 bots
Archivers and validators
Archives saving a copy of a page and tools checking its markup.
16 bots
Other bots
Feed readers, security scanners, research crawlers and the long tail.
641 bots