Who's crawling your site?

AI crawlers

AI companies run several kinds of bot: crawlers that gather training data, crawlers that build an AI search index, and fetchers that load a page because someone asked an assistant about it. Each one usually has its own robots.txt token, so you can allow one and block another. Vireo counts the pages these bots take and shows them next to the readers AI assistants send you. Here's how that looks.

About this data

Bot names, operators, types, user agents and patterns come from Matomo Device Detector, released under the GNU LGPL v3. Vireo Analytics' own bot signatures are built from the same file. Details on AI crawlers, including their robots.txt tokens, come from the ai.robots.txt project (MIT licence). Where an operator documents its crawler, the page quotes that documentation and links to it; those pages were checked on 2026-10-06.

Each bot's category is what Vireo Analytics' bot filter actually returns for that bot's user agent, run against the plugin's code. Bots the filter doesn't recognise aren't listed. Facts that no source states, such as whether an obscure crawler runs JavaScript, are left out rather than guessed.

Device Detector commit e83ce7a, ai.robots.txt commit 9ad8a47.