AI crawler

Three kinds of AI crawler

AI companies run different bots for different jobs, and the difference decides whether blocking one costs you anything. Training crawlers, such as GPTBot, ClaudeBot and CCBot, collect pages for model training. They send no readers. Search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, build the index an AI search product answers from, and those answers can link back to you. User fetchers, such as ChatGPT-User, Claude-User and Perplexity-User, request a page because a person asked an assistant about it.

Most operators give each kind its own robots.txt token, so you can refuse training and still appear in AI answers. Some user-triggered fetchers don't follow robots.txt at all, because a person asked for the page. Each bot's page in the directory says what its operator documents.

How Vireo counts them

AI crawlers are one of the bot categories Vireo keeps out of your pageviews and visitors. Their requests are counted when they ask WordPress for a page, so crawlers that never run JavaScript are still seen.

The AI assistants card on the Search & AI tab sets three numbers side by side for the same days: requests from AI crawlers, readers AI assistants sent back, and crawler requests per reader sent. If Vireo started counting crawlers part-way through the range, the card starts at that date and says so, because earlier days have readers but no crawler figures and would make the ratio look smaller than it is.

Pitfalls

Full-page caching hides crawler requests. A cached page is answered before WordPress runs, so on a cached site the real number of AI crawler requests is higher than the card shows.

On a headless site the count covers requests to WordPress only. Crawlers that read your front end never reach it.

Blocking a crawler in robots.txt doesn't remove what it already collected, and only works on crawlers that honour the file. The robots.txt generator on this site builds the rules for the tokens you choose.