Robots.txt generator
Build a robots.txt file that tells search engines and AI crawlers what they may crawl. Start from a preset, fine-tune the rules, test any URL against them, then copy or download the file.
- AI crawlers listed by purpose
- URL tester with Google's matching rules
- Runs in your browser
Used for the Sitemap line and the upload address.
Your robots.txt
Open to all crawlers
- Start fromAllow allPreset
- Rules—Paths and bots
- Sitemap—Helps discovery
- Upload to/robots.txtRoot of your host
Build your robots.txt
All crawlers
The User-agent: * group. Every bot without its own group follows it.
Common blocks
AI crawlers
Checked bots get Disallow: /. Blocking training bots does not affect normal search.
Training crawlers
AI search crawlers
Fetches a user asked for
Rules for one bot
A bot with its own group ignores the All crawlers group, so its rules are copied in unless you turn that off.
No bot groups yet.
Sitemaps
Full URLs, one per line. Filled in from your website address until you edit it.
Test a URL
Checks the file above the way Google reads it.
What robots.txt does
A plain text file at the root of a host that tells crawlers which URLs they may request.
Groups and rules
Each group starts with one or more User-agent lines, followed by Disallow and Allow rules. A bot follows only the most specific group that names it.
The longest rule wins
Google compares every rule that matches a URL and uses the longest one. When an Allow and a Disallow are equally long, Allow wins. * matches anything, $ marks the end.
A request, not a lock
Search engines and most AI companies respect robots.txt, but nothing forces a bot to. Anyone can still open a blocked URL, so never use it to hide private pages.
One file per host
The file must be at /robots.txt on each host and protocol, and Google reads up to 500 KiB of it. Paths are case-sensitive.
robots.txt, noindex or a login?
Three tools for three different jobs. Using the wrong one is the most common mistake.
| Method | Stops crawling | Keeps it out of search | Protects private content |
|---|---|---|---|
robots.txt Disallow |
Yes, for bots that obey it | No, the URL can still be listed from links | No |
noindex meta tag or header |
No, the page must be crawled to read it | Yes | No |
Password or login |
Yes | Yes | Yes |
404 or 410 status |
The page is gone | Removed over time | Content is deleted |
How to use this generator
From a preset to a file on your server in four steps.
Pick a starting point
Allow all, WordPress, Block AI training or Block all fills the builder with a sensible setup. Your website address adds the sitemap line.
Adjust the rules
Add paths, file types, AI crawlers and groups for single bots. The preview, checks and tester update as you type.
Test important URLs
Type a path and pick a bot to see whether it may crawl it, and which line decides, using Google's matching rules.
Upload and verify
Download robots.txt, put it at the root of your host, open it in a browser, then check the robots.txt report in Search Console.
What this tool does, and what it doesn't
A plain list, so you know exactly what to expect.
This tool does
- Build User-agent, Allow, Disallow, Crawl-delay and Sitemap lines
- List AI crawlers by training, search and user fetches
- Copy shared rules into bot-specific groups
- Test URLs with Google's longest-match rule
- Warn about blocked CSS, JS and relative sitemaps
- Remember your setup in this browser
This tool does not
- Upload the file to your server
- Read your live robots.txt
- Force bots to obey the rules
- Remove pages from search results
- Protect private content
- Send your setup to our server
Where your data goes
Nowhere outside your browser. The file is generated on this page and your setup is saved in your browser's local storage so a reload keeps it. Reset clears it. Copying and downloading happen on your device. Like any website, our server keeps normal access logs of page visits, as described in our privacy policy.
Step-by-step guide by Bhavin J. Sheth.
Problems after uploading
The six issues people run into most, and what each one means.
robots.txt returns 404
The file is not at the root. It must load at https://example.com/robots.txt. A file in a subfolder is ignored.
Google still uses the old rules
Google caches robots.txt for up to 24 hours. Request a recrawl in the robots.txt report in Search Console.
"Indexed, though blocked by robots.txt"
Google found links to a blocked page. To remove it from results, allow crawling and add noindex instead.
Traffic dropped after upload
Often a staging file with Disallow: / went live. Use the tester on your key pages before uploading.
A subdomain ignores the rules
Every host needs its own file. blog.example.com reads blog.example.com/robots.txt, not the main one.
AI bots still visit
robots.txt is voluntary and some agents do not identify themselves. Block them at your server or CDN as well.
Questions about robots.txt
Do I need a robots.txt file?
No. Without one, crawlers assume they may visit every page. A robots.txt file is useful when you want to keep crawlers out of areas such as admin pages, internal search results or cart pages, block specific bots, or point crawlers to your sitemap. If your file is missing or returns 404, Google crawls everything.
Does robots.txt keep a page out of Google?
No. It stops crawling, not indexing. If other pages link to a blocked URL, Google can still list it, without a description. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header, or protect the page with a login.
Should I block AI crawlers?
It depends on what you want. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models, while search and user agents such as OAI-SearchBot or ChatGPT-User fetch pages so AI assistants can show and link to you. Blocking Google-Extended stops use for Gemini training but does not remove your pages from Google Search. Robots.txt is voluntary, so some bots may ignore it.
Why does a bot with its own group ignore my other rules?
A crawler follows only the most specific group that names it. If you add a group for Googlebot, Googlebot ignores the User-agent: * group completely. That is why this generator can copy the rules for all crawlers into each bot group, so the bot keeps following them too.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl rate to how fast your server responds. Bing and some other crawlers do respect it. If Googlebot is overloading your server, return 503 or 429 responses for a short time, and fix the slow pages.
Where do I upload the file on WordPress?
The file must be at the root of the host, such as https://example.com/robots.txt, and each subdomain needs its own. WordPress shows a virtual robots.txt when no file exists. You can paste the generated text into the robots.txt editor of Yoast SEO or Rank Math, or upload a robots.txt file to the folder that holds wp-config.php, which then replaces the virtual one.
Is my configuration sent to your server?
No. The file is built in your browser and nothing is sent to our server. Your current setup is saved in your own browser so a reload does not lose it, and you can clear it with Reset. Like any website, our server keeps normal access logs of page visits, as described in our privacy policy.