Crawler directory · SEO audit crawler
SiteCheckerBotCrawler: user agent, robots.txt rules, and how to allow or block it
SiteCheckerBotCrawler is the crawler behind Sitechecker’s SEO audit tool. It is not an AI crawler: it visits your site when you, or someone auditing you, runs a Sitechecker scan, and it checks things like status codes, titles and links.
- Operator
- Sitechecker
- Feeds
- Sitechecker SEO audits
- Type
- SEO audit crawler
- robots.txt token
- SiteCheckerBotCrawler
SiteCheckerBotCrawler user agent
Requests from SiteCheckerBotCrawler carry a User-Agent header like this (example — vendors change version numbers, so match on the token SiteCheckerBotCrawler, not the whole string):
Mozilla/5.0 (compatible; SiteCheckerBotCrawler/1.0; +http://sitechecker.pro)
To see how often it visits, filter your access log with grep -i "SiteCheckerBotCrawler" access.log. A User-Agent can be faked, so treat it as a claim, not proof.
Should you allow or block SiteCheckerBotCrawler?
Blocking stops that tool auditing your site. It has no effect on Google rankings or on AI visibility.
Heads up: If you are auditing your own site with Sitechecker, make sure your robots.txt and firewall are not blocking this token, or the audit will report false errors.
robots.txt rules for SiteCheckerBotCrawler
Put these in the robots.txt at the root of your domain.
Block SiteCheckerBotCrawler
User-agent: SiteCheckerBotCrawler Disallow: /
Explicitly allow SiteCheckerBotCrawler
User-agent: SiteCheckerBotCrawler Allow: /
Block only part of your site
User-agent: SiteCheckerBotCrawler Disallow: /private/ Allow: /
robots.txt states your policy; it does not enforce it. A CDN or firewall can block a bot your robots.txt allows, which is why it is worth testing rather than assuming.
Check what your site does today
The free AI bot checker reads your robots.txt and tells you whether GPTBot, ClaudeBot, PerplexityBot and nine other AI crawlers are allowed, blocked or crawling by default, then runs a live GPTBot fetch to prove your content is actually served.
Check my site free →Questions, answered
What is SiteCheckerBotCrawler?+
SiteCheckerBotCrawler is the crawler behind Sitechecker’s SEO audit tool. It is not an AI crawler: it visits your site when you, or someone auditing you, runs a Sitechecker scan, and it checks things like status codes, titles and links.
How do I block SiteCheckerBotCrawler in robots.txt?+
Add these two lines to the robots.txt at the root of your domain: User-agent: SiteCheckerBotCrawler Disallow: / User-agent tokens are matched case-insensitively. The rule applies from the crawler's next visit; it does not delete anything already collected. If you are auditing your own site with Sitechecker, make sure your robots.txt and firewall are not blocking this token, or the audit will report false errors.
Will blocking SiteCheckerBotCrawler affect my Google rankings?+
No. SiteCheckerBotCrawler is a third-party audit crawler and has no bearing on Google rankings.
What does the SiteCheckerBotCrawler user agent look like?+
An example User-Agent header is: Mozilla/5.0 (compatible; SiteCheckerBotCrawler/1.0; +http://sitechecker.pro). Vendors change version numbers, so match on the token SiteCheckerBotCrawler rather than the whole string. A User-Agent can be spoofed, so verify important traffic against the vendor's published details before trusting it.