Create a robots.txt File for Your Website
Generate crawler directives for search engines. Control which pages and bots can access your site, add your sitemap URL, and download a ready-to-upload file.
What Is a robots.txt File?
A plain text file placed at the root of your website that tells search engine crawlers which pages or files they may or may not request.
A Guide for Crawlers, Not a Firewall
robots.txt uses directives like User-agent, Allow, Disallow, and Sitemap to give instructions to crawlers. It is a request, not an enforcement mechanism — malicious bots may ignore it entirely.
Why Use One?
It can help direct crawl budget toward important content, discourage crawling of admin areas or duplicate content, and point crawlers to your sitemap.
What It Cannot Do
It does not prevent pages from being indexed if other sites link to them. For true access control, use authentication or noindex meta tags.
Placement Matters
The file must live at the root of your domain (https://yourdomain.com/robots.txt) and be served with a 200 HTTP status. Subdirectory robots.txt files are ignored.
Robots.txt Generator
1. General Settings
Most major search engines ignore this. Useful for smaller bots.
2. General Allow / Disallow Rules
3. Bot-Specific Rules
4. Sitemap
Auto-generated from Website URL, or edit manually. If you need to create a sitemap, try our XML Sitemap Generator.
Live Preview
Save & Load Configurations
Privacy note: your configuration is generated entirely in your browser. Nothing is sent to our server.
What This Robots.txt Generator Builds
Every field maps to a real directive that will appear in your output file.
Live Preview
The file updates in real time as you type — you see exactly what will be uploaded to your server.
Bot-Specific Rules
Create custom User-agent blocks for Googlebot, Bingbot, or any other crawler, overriding the default rules.
Pre-built Templates
Start from "Allow All", "Block All", or "WordPress Default" — one-click starting points for common setups.
Sitemap Integration
Adds the Sitemap: directive automatically from your website URL.
Copy or Download
Copy the content to clipboard or download as a ready-to-upload robots.txt file.
Save Configurations
Store complex setups in your browser's local storage and reload them on future visits.
Honest by Design
This tool builds robots.txt directives. Here is exactly what it does and does not do:
- ✓ Generates correctly formatted robots.txt directives from your inputs.
- ✓ All generation happens in your browser — your configuration is not sent to our server.
- ✓ Does not upload the file for you — you still upload it to your website's root directory yourself.
- ✓ Does not guarantee search engines will follow your directives — compliant bots do, malicious bots may not.
- ✓ Does not replace true access control — for private content, use authentication or
noindex.
This tool does
- Generate correctly formatted robots.txt directives
- Support User-agent, Allow, Disallow, Crawl-delay, Sitemap
- Create bot-specific rule blocks
- Provide one-click templates
- Run entirely in your browser
- Save/load configurations from local storage
This tool does not
- Upload the file to your server
- Validate whether your directives are correct for your site
- Force crawlers to obey the rules
- Replace authentication for private content
- Prevent indexing if other sites link to your URLs
- Test whether your live robots.txt is reachable
robots.txt vs noindex vs Authentication
Three different tools for three different problems. Using the wrong one is the most common mistake.
robots.txt Disallownoindex meta tagThe common mistake
Blocking a URL in robots.txt does not remove it from Google's index. If other sites link to that URL, Google can still show it in search results (without knowing what's on the page). To actually remove a page from search results, use noindex — but the page must be crawlable for Google to read that tag.
Want to Understand robots.txt & Search Engine Crawling?
We published a complete guide explaining what robots.txt is, how crawlers use it, common directives like Allow, Disallow, and Sitemap, SEO best practices, common mistakes to avoid, and how to generate a robots.txt file for free.
📘 Read Full robots.txt Guide →How to Use This Generator
Four steps from configuration to uploading your file.
Set Defaults
Choose a template or set your default User-agent: * rules.
Add Custom Rules
Add specific rules for particular bots or disallow particular directories and file types.
Add Your Sitemap
Enter your website URL to auto-generate the sitemap directive, or enter the full sitemap URL manually.
Copy or Download
Copy the content or download robots.txt, then upload it to your website's root directory.
Common Issues After Upload
If crawlers aren't behaving as expected after uploading robots.txt, check these common causes.
File Not Found (404)
UPLOAD ERRORCrawlers cannot reach https://yourdomain.com/robots.txt.
Solution: Upload the file to the root directory of your website, not a subdirectory. Verify by visiting the URL in your browser.
Bots Still Crawling Blocked URLs
DIRECTIVE IGNOREDSome bots ignore robots.txt entirely, or other sites link to the blocked URLs.
Solution: Use noindex meta tags or password protection for truly private content. robots.txt is a request, not a block.
Pages Dropped from Index
OVER-BLOCKINGToo many Disallow rules may have blocked important content.
Solution: Review your rules. Use Google Search Console's robots.txt tester to verify you haven't blocked pages you want indexed.
Rules Not Taking Effect
CACHE DELAYSearch engines cache robots.txt and don't refresh immediately.
Solution: Wait 24–48 hours for crawlers to re-fetch the file. You can also request a refresh in Google Search Console.
Sitemap Not Detected
MISSING DIRECTIVESearch engines haven't found your sitemap via robots.txt.
Solution: Ensure the Sitemap: line is present and the URL returns a 200 response. Also submit the sitemap directly in Search Console.
Private Content Indexed
WRONG APPROACHYou relied on robots.txt alone to protect private areas.
Solution: robots.txt does not secure content. Use authentication, HTTP status codes (401/403), or noindex for pages that must stay out of search results.
Common Use Cases — What to Block and What to Allow
Use the generator for these common scenarios.
Disallow: /admin/, /wp-admin/, /private/Crawl-delay for aggressive bots, Disallow heavy file types.Sitemap: https://yourdomain.com/sitemap.xmlUser-agent: AhrefsBot + Disallow: /User-agent: Googlebot-Image + specific Allow rules.Robots.txt Generator FAQs
Answers based strictly on what this tool does and how robots.txt works.
Is a robots.txt file required?
No — a website works without one. But it can help guide crawler access and manage crawl behavior on larger or more complex websites, which is valuable for SEO.
What is the "Default User-agent"?
User-agent is the name of a crawler (Googlebot, Bingbot, etc.). The wildcard * means "all bots". Rules under the default user-agent apply to every crawler unless a more specific rule overrides them.
What is Crawl-delay?
A directive telling crawlers to wait a number of seconds between requests. Useful for slow or shared servers. Major search engines like Google ignore it, but many smaller bots respect it.
What do Allow and Disallow do?
Disallow tells a bot not to access a directory, file, or file type. Allow creates an exception — for example, blocking /images/ entirely but allowing /images/logo.png.
What are bot-specific rules?
Rules that apply only to a specific user-agent. They override the default * rules — for example, blocking /secret/ for all bots but allowing Googlebot to access it.
Can robots.txt prevent indexing?
No. It only discourages crawling. If other sites link to a blocked URL, search engines can still index it. For true removal from search results, use noindex meta tags or authentication.
Where do I upload the file?
At the root of your domain: https://yourdomain.com/robots.txt. Files in subdirectories are ignored. Verify by visiting the URL directly in your browser.
Is my configuration sent to your server?
No. All generation happens in your browser. Your configuration is never sent to our server. Only the downloaded file leaves your browser.