Robots.txt Tester
Test whether Googlebot, Bingbot and AI crawlers such as GPTBot and PerplexityBot can crawl any URL, see the exact matching rule and fix syntax errors.
What this tool checks
Every result comes with a clear explanation and a prioritised fix, so you know exactly what to do next.
Google and Bing
Tests Googlebot, Googlebot-Image, Bingbot, DuckDuckBot and YandexBot.
AI crawlers
GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and more.
Matching rule
Shows the exact Allow or Disallow line that decides each result.
Syntax errors
Invalid lines, rules outside a group and unsupported directives.
Sitemap lines
Checks that every listed sitemap actually loads.
Risky blocks
Flags blocking the whole site or CSS/JS files.
Three steps to better results
No account, no API key and no limits. Just enter your details and get answers in seconds.
Choose a source
Fetch a live robots.txt or paste a draft.
Enter a URL path
The page or folder you want to test.
Read the verdict
Allowed or blocked for every crawler, with the rule.
Frequently asked questions
Quick answers about the Robots.txt Tester.
Still have a question?
Email us and a real person replies within one business day.
What is robots.txt?
A text file at the root of your site that tells crawlers which URLs they may crawl. It controls crawling, not indexing.
Does blocking a page in robots.txt remove it from Google?
No. A blocked URL can still be indexed if other pages link to it. To keep a page out of search, allow crawling and add a noindex tag.
How does Google pick which rule applies?
Google uses the most specific user-agent group, then the longest matching rule. If Allow and Disallow match with equal length, Allow wins. This tester follows the same logic.
Should I block AI crawlers?
It depends. Blocking search bots (OAI-SearchBot, PerplexityBot) removes you from AI answers. Blocking training bots (GPTBot, Google-Extended, CCBot) opts you out of model training while keeping AI search visibility.
Can I test changes before publishing?
Yes. Use “Paste robots.txt” to test a draft against any path without touching your live site.
Want us to fix it for you?
Our team turns these results into rankings. Book a free strategy call and get a clear, prioritised SEO plan for your website.
Related SEO tools
Robots.txt Generator
Build a correct robots.txt in seconds with presets for WordPress and other platforms, sitemap lines and simple controls…
- Technical
XML Sitemap Checker
Find and validate your XML sitemap: XML errors, child sitemaps, URL count, lastmod dates, duplicates and a live…
- Technical