Meta-WebIndexer

Operator

Meta Platforms, Inc.

Type

Social Media Agent

Official page

developers.facebook.com

robots.txt token

meta-webindexer

In Vireo

Excluded, reported under Link previews

What it does

As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

ai.robots.txt describes its function as: AI Assistants

Source: ai.robots.txt.

Crawl frequency, per ai.robots.txt: Unhinged, more than 1 per second.

User agent

Requests from Meta-WebIndexer carry a user agent like this:

meta-webindexer

Device Detector identifies it with this pattern (a case-insensitive regular expression):

meta-webindexer

Anyone can put any name in a user agent. Where the operator publishes IP ranges, checking the request's address against them is the way to confirm a visit is genuine.

How to block or allow Meta-WebIndexer

Add a group for its token to the robots.txt file at the root of your site. To keep it out of the whole site:

User-agent: meta-webindexer
Disallow: /

To let it crawl everything, use an empty Disallow: or Allow: / instead. WordPress serves a virtual robots.txt that many SEO plugins let you edit. A real robots.txt file uploaded to the site root takes its place.

robots.txt is a request, so it only works on crawlers that honour it. A bot that ignores it has to be blocked at the server or firewall, by user agent or IP address.

How Vireo counts Meta-WebIndexer

Vireo keeps requests from Meta-WebIndexer out of your pageviews and visitors and counts them on its bot report under Link previews, so you can see how much it left out rather than wonder why your server logs show more traffic.

Checked by running Vireo Analytics' own bot filter over Meta-WebIndexer's user agents.

Sources

• Meta-WebIndexer information page (developers.facebook.com)
• Matomo Device Detector, bots.yml (LGPL-3.0)
• ai.robots.txt (MIT)