AI companies read websites all day, and the people who run those sites mostly can’t see it happening. Your analytics shows the readers who arrive from ChatGPT or Perplexity. It doesn’t show the crawlers that read your pages on the way to those answers.

The two numbers belong together. One is what the AI companies take from your site. The other is the readers their assistants send back. Comparing them is how you find out whether AI is a traffic source for your site or mostly a cost.

Three kinds of AI crawler

The big AI companies don’t run one crawler each. They run several, and each does a different job.

A training crawler collects pages to train future models. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the best known. What they take ends up in a model months later, with no link back to you.

A search crawler builds the index an assistant searches when it answers a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this. Being in that index is what lets an assistant cite your page and link to it.

A user fetcher reads one page live, because someone asked a question and the assistant decided your page would help. ChatGPT-User, Claude-User and Perplexity-User work this way. Each fetch is a real person’s question being answered with your content, usually without that person opening your page.

Google works differently. Much of its AI reading is done by the same Googlebot that crawls for Search, so you can’t block Google’s AI by crawler name without leaving Search too. Google-Extended is a robots.txt token rather than a crawler. It controls whether your content is used to train Gemini and ground its answers, and Google says it has no effect on Search.

The bot directory lists the AI crawlers Vireo recognises, with what each one is for.

Why your analytics can’t see them

Most analytics tools count a visit when a small script runs in the reader’s browser. Crawlers fetch the page’s HTML and move on. They almost never run JavaScript, so a script-based tracker never hears about them. That’s why the AI crawlers reading your site don’t show up in your visitor numbers, and also why you can’t tell how many there are.

To count them, you have to catch the request on the server, when the crawler asks for the page. Vireo does that inside WordPress. It recognises over 800 named bots by their user agent, files each request under a category such as AI crawlers or search engines, and keeps all of them out of your visitor and pageview totals. How much of your traffic is bots covers the rest of that filtering.

The other half is in your referrers. When someone clicks a link inside ChatGPT, Gemini or Claude, the visit usually arrives with the assistant’s address as its source. Vireo groups those visits by assistant, so ChatGPT and Perplexity each get a line instead of disappearing into a long referrer list.

See your own ratio in Vireo

In WordPress, open Vireo Analytics and switch to the Search & AI tab. The AI assistants card shows three figures for the date range you’ve picked:

  • Pages their crawlers took: the requests from AI crawlers that the bot filter held back.
  • Readers they sent back: visits that arrived by following a link inside an assistant.
  • Pages taken per reader returned: the first divided by the second.

Under them, a table lists how many readers each assistant sent. The card appears once there’s something to show, so a brand-new install may not have it yet.

Pick a range of a few weeks or more. Referrals from assistants are small numbers on most sites, and a single week jumps around too much to read.

A large number in the third box tells you the AI companies are reading far more of your site than they’re sending readers to. Whether that’s a problem depends on what you want from AI, and the card shows you the size of the trade.

What it counts and what it misses

The crawler figure is counted inside WordPress, so a page served from a full-page cache never reaches it. On a heavily cached site the real number of AI crawler requests is higher than the card shows, which makes the ratio worse, not better.

The readers figure only counts visits that arrived with the assistant as their source. Some assistants and apps strip that information, and someone who reads an answer and then types your address in arrives as direct traffic. So treat it as the floor for what assistants send.

Your robots.txt choices, and what each costs

The crawlers from OpenAI, Anthropic and Perplexity read robots.txt, the plain-text file at the root of your site that tells crawlers what they may fetch. Each crawler has its own name in the file, so you can allow one job and refuse another.

Blocking training crawlers keeps your new content out of future models. It costs you almost nothing in visits, because training crawlers never send readers anyway. This is the cheapest choice to make:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Blocking search crawlers takes you out of the assistants’ search indexes. Your pages stop being cited, and the readers who would have followed those citations stop coming. If assistants send you anything worth keeping, this is the choice that costs it.

Blocking user fetchers is the least reliable. Anthropic says its crawlers, Claude-User included, respect robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. For those two, a robots.txt rule may not stop the fetch. Stopping them for certain means blocking at the server or the CDN, by user agent or by the IP ranges the companies publish.

Allowing everything is also a choice. It keeps you in every index and every live answer, and the AI assistants card tells you what that’s earning.

WordPress builds a robots.txt for you if there isn’t a file on disk. To change it, either upload your own robots.txt to the site’s root folder, or use the robots.txt editor in your SEO plugin if it has one. After you change it, open yoursite.com/robots.txt in a browser and check the rules are there.

Check whether a block worked

A block shows up in the card’s first figure. Leave it a couple of weeks after the change, then compare the same length of range before and after. The crawlers you blocked should fall away. If one keeps fetching, it’s either ignoring the file or using a name you didn’t block. The ChatGPT-User page and the rest of the bot directory show the exact names to use.

To check how Vireo classes a particular crawler, go to Vireo Analytics, Settings, Tracking, and use the Bot filter test. Paste the user agent from your server logs and it tells you whether Vireo counts it and which category it falls under. The bot filtering docs cover the rest.