AI crawler robots.txt generator
Tick the AI crawlers you want to keep off your site and copy the robots.txt lines. They're grouped by what each one does with your pages, because blocking a crawler that takes content costs nothing, and blocking one that cites you can cost readers.
Take or send
AI companies read your site with different crawlers for different jobs. A training crawler takes your pages into a model with no link back. A search crawler and a user fetcher can lead to an answer that cites your page, and some of those readers click through. The post on what AI crawlers take and what AI assistants send back covers the trade-off for each robots.txt choice.
Where the tokens come from
Every token here is in the data behind the bot directory. 16 of them are quoted from the operator's own crawler documentation, and they're listed first in each group. The rest come from the ai.robots.txt project. Crawlers without a known token aren't listed, because a guessed name in robots.txt blocks nothing.
What robots.txt can't do
robots.txt is a request. Crawlers that honour it stop fetching the pages you disallow, and the rest carry on. Stopping those means blocking at the server or the CDN, by user agent or by the IP ranges some operators publish. Blocking a crawler also doesn't remove what it already took.
WordPress serves a virtual robots.txt when there's no file on disk, and many SEO plugins let you edit it. After a change, open yoursite.com/robots.txt to check the lines are there. With Vireo Analytics, you can watch the crawler requests fall away over the following weeks, and check any user agent with the bot checker.
See what AI crawlers take from your site
Vireo Analytics is a free WordPress plugin. It counts AI crawler requests and sets them against the readers AI assistants send back.
Get Vireo free on WordPress.org