Robots.txt Generator
Build a correct robots.txt in seconds with presets for WordPress and other platforms, sitemap lines and simple controls for AI crawlers.
What this tool checks
Every result comes with a clear explanation and a prioritised fix, so you know exactly what to do next.
Platform presets
Safe defaults for WordPress, generic sites and staging.
Disallow and Allow
Block admin, search and cart pages with exceptions.
Sitemap lines
Adds your sitemap URLs in the correct absolute format.
AI crawler policy
Allow all, block training only, or block all AI bots.
Crawl-delay
Optional delay for Bing and Yandex (Google ignores it).
Safety checks
Warns about blocking your whole site or CSS/JS files.
Three steps to better results
No account, no API key and no limits. Just enter your details and get answers in seconds.
Pick a preset
Start from WordPress or a generic template.
Adjust the rules
Add paths, sitemaps and your AI crawler policy.
Copy or download
Upload robots.txt to the root of your site.
Frequently asked questions
Quick answers about the Robots.txt Generator.
Still have a question?
Email us and a real person replies within one business day.
Where do I put robots.txt?
In the root of your domain so it loads at https://yourdomain.com/robots.txt. In WordPress you can edit it in Rank Math or Yoast settings.
What should a WordPress robots.txt include?
Usually just Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php, plus your sitemap line. Do not block /wp-content/ or /wp-includes/, because Google needs those files to render pages.
What does "block AI training, allow AI search" do?
It blocks crawlers used to train AI models (GPTBot, ClaudeBot, Google-Extended, CCBot and others) while allowing the crawlers that power AI search answers, so you can still be cited.
Does Google follow Crawl-delay?
No. Google ignores it and manages crawl rate automatically. Bing and Yandex respect it.
Can robots.txt hide private pages?
No. It is public and only a request to well-behaved crawlers. Protect private content with a login and use noindex to keep pages out of search.
Want us to fix it for you?
Our team turns these results into rankings. Book a free strategy call and get a clear, prioritised SEO plan for your website.
Related SEO tools
Robots.txt Tester
Test whether Googlebot, Bingbot and AI crawlers such as GPTBot and PerplexityBot can crawl any URL, see the…
- Technical
XML Sitemap Checker
Find and validate your XML sitemap: XML errors, child sitemaps, URL count, lastmod dates, duplicates and a live…
- Technical
SEO Analyzer
Run a full on-page, technical, content, AEO and GEO audit of any page in seconds, with a clear…
- Audit