archive.org bot

Operator

Internet Archive (Archive-It)

Type

Crawler

Official page

support.archive-it.org

Follows robots.txt

Partly, according to Internet Archive (Archive-It).

In Vireo

Excluded, reported under Archivers and validators

What it does

The Internet Archive crawler that Archive-It uses to collect sites for its partners' web archives.

The name (user-agent) of our crawler is archive.org_bot.

Internet Archive (Archive-It)

User agent

Requests from archive.org bot carry user agents like these:

Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)

Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)

Device Detector identifies it with this pattern (a case-insensitive regular expression):

archive\.org_bot|special_archiver

Anyone can put any name in a user agent. Where the operator publishes IP ranges, checking the request's address against them is the way to confirm a visit is genuine.

Worth knowing

Archive-It customers can add an 'Ignore Robots.txt' rule for a seed or host, so robots.txt isn't guaranteed to block it. No primary source found for ia_archiver, and the Internet Archive's own Wayback crawl policy page wasn't found.

If contacting the site owner is not an option, you can add an Ignore Robots.txt scoping rule to the seed Seed or Host (collection level).

Internet Archive (Archive-It)

How Vireo counts archive.org bot

Vireo keeps requests from archive.org bot out of your pageviews and visitors and counts them on its bot report under Archivers and validators, so you can see how much it left out rather than wonder why your server logs show more traffic.

Checked by running Vireo Analytics' own bot filter over archive.org bot's user agents.

Sources

• Internet Archive (Archive-It): crawler documentation (checked October 2026)
• archive.org bot information page (archive.org)
• Matomo Device Detector, bots.yml (LGPL-3.0)