> robots | crawl | seo <
// Generate a valid robots.txt file with presets, sitemap, bot toggles and per-user-agent rules
QUICK BOT TOGGLES
// Toggle a checkbox to add a Disallow: / rule for that crawler. Useful for AI scrapers.
CMS Presets + AI Bots
One-click templates for WordPress, Shopify, Ghost plus quick toggles to block AI scrapers like GPTBot, ChatGPT-User, CCBot, Claude-Web, and anthropic-ai.
Per User-Agent Rules
Add as many rule blocks as you need. Each block accepts multiple Allow and Disallow paths plus an optional crawl-delay directive.
Live Preview, Copy or Download
The output pane updates as you type. Copy to clipboard or download as robots.txt with a single click — 100% browser-based, no server.
// ABOUT ROBOTS.TXT
How robots.txt Works:
robots.txt is a plain-text file at the root of your site that follows the Robots Exclusion Protocol (RFC 9309). Crawlers read it before fetching pages and respect the Allow / Disallow rules associated with their user-agent. The file is advisory, not a security boundary.
Example:
User-agent: * Disallow: /admin/ Allow: / Sitemap: https://example.com/sitemap.xml
Standards & References:
- >RFC 9309: Robots Exclusion Protocol (IETF, 2022)
- >Google's robots.txt documentation (developers.google.com/search)
- >Sitemaps protocol (sitemaps.org)
- >Wildcard support (* and $) widely accepted by major crawlers
Common Use Cases:
- >Block crawlers from staging or admin sections
- >Point search engines at your sitemap
- >Reduce crawl load with crawl-delay directives
- >Tailor rules per bot (Googlebot, Bingbot, AhrefsBot, GPTBot)
- >Standardize robots.txt across CMS deployments
>> frequently asked questions
Q: Where do I place the generated robots.txt?
A: Upload it to the root of your domain so it is served at https://yourdomain.com/robots.txt. Subdirectory placements are not respected by crawlers — the file must be reachable at exactly that path. For multi-host setups, every hostname (subdomains included) needs its own robots.txt; rules do not cascade across hostnames.
Q: Does Disallow protect content from being seen?
A: No. robots.txt is advisory and is in fact a public file that anyone can read, so it can even leak the existence of sensitive paths. Compliant crawlers obey it, but malicious bots and casual visitors can ignore it entirely. For genuine privacy, use authentication, IP allow-lists, or noindex meta/HTTP headers — robots.txt is purely a politeness convention.
Q: Does Google honor crawl-delay?
A: Googlebot ignores the crawl-delay directive entirely. Use the Search Console crawl-rate setting instead, or rely on Google's adaptive crawler. Bing, Yandex, Seznam, and several smaller engines do honor crawl-delay, so the directive is still useful for non-Google traffic shaping if your origin is bandwidth-constrained.
Q: Can I have multiple Sitemap entries?
A: Yes. The Sitemap directive can appear multiple times and is independent of user-agent blocks — list each sitemap URL on its own line, anywhere in the file. Sitemap URLs must be absolute and may live on a different host than robots.txt itself, provided that host is verified in your search console.
Q: What is the syntax for wildcards?
A: Most major crawlers (Googlebot, Bingbot, Yandex) support * (any character sequence) and $ (end of URL). Example: Disallow: /*.pdf$ blocks every URL ending in .pdf. RFC 9309 makes wildcard support optional, so older or smaller crawlers may not honor them — always test critical rules in Search Console's robots.txt tester.
Q: How do I block AI training crawlers?
A: Add explicit Disallow rules for the user-agents OpenAI uses (GPTBot, ChatGPT-User), Common Crawl (CCBot), Anthropic (Claude-Web, anthropic-ai), Google AI (Google-Extended), and Perplexity (PerplexityBot). The bot toggles in this generator add Disallow: / blocks for each selected agent. Note that compliance is voluntary; some scrapers ignore robots.txt outright.