SEO tool crawlers

Crawlers run by SEO and backlink tools to map links between sites and audit pages. They aren't readers, and some of them crawl often.

17 known bots from 9 operators. 13 have their own page with user agents, robots.txt tokens and sources.

BotOperatorType
aHrefs BotAhrefsCrawler
BarkrowlerNot knownCrawler
BLEXBot CrawlerWebMeUpCrawler
DataForSeoBotNot knownCrawler
DotBotMozCrawler
LinkpadBotSolomono LLCCrawler
MegaIndexNot knownCrawler
MJ12 BotMajesticSearch bot
NeevabotNeeva Inc.Search bot
RogerbotMozCrawler
Screaming Frog SEO SpiderScreaming Frog LtdCrawler
SemrushBotSemrushCrawler
SemrushBot-OCOBSemrushListed by ai.robots.txt
SemrushBot-SWASemrushListed by ai.robots.txt
serpstatbotSerpstatCrawler
SiteAuditBotSemrushCrawler
ZoominfoBotNot knownCrawler

Names, operators and types from Matomo Device Detector (LGPL-3.0) and ai.robots.txt (MIT). Category from running Vireo Analytics' bot filter on each bot's user agent. About this data.

Not sure what a request in your logs is? Paste its user agent into the bot checker.

Other categories