Free · Training vs search bots · Checker included

AI Robots.txt Generator: Block AI Training, Stay in AI Search

Choose which AI bots may visit your site, from GPTBot and ClaudeBot to Perplexity and Google-Extended. Keep your current rules, copy a ready robots.txt, and check what any website tells each AI bot.

  1. Choose the AI bots
  2. Paste your current file
  3. Copy and add to your site

By Updated

Choose what AI bots may do

Quick choice
Bot by bot (0 AI bots)

Rules for AI bots in it are replaced; everything else stays as it is.

Your robots.txt

0 blocked0 allowed

        

          
          
        

        

          
Good to know

          
    
        
    
      
    
    
    
  
    

    



    
  
    
    
    Checker
    Which AI bots does a website allow?
    Enter any website, or paste a robots.txt, and see what each AI bot is told.
    
    
    
      
    
        
    
          
    
          
        
    
        
    Or paste a robots.txt
    
          
    
          
    
        
    
        
    
        
        
      
    
    
    
  
    

    



    
  
    
    
    Honest by design
    What robots.txt can and can't do
    
    
    
      
    What it does
      
        
    • Asks AI bots politely to stay out of your site or certain folders. Major companies such as OpenAI, Anthropic, Google, Apple and Perplexity say their main bots follow it.
      • 
        
    • Separates training from search, so you can keep appearing in AI answers while saying no to model training.
      • 
        
    • Keeps your other rules when you paste your current file.
      • 
        
    • Shows exactly what each bot is told, using the same matching rules crawlers use.
      • 
      
    
      
    What it doesn't do
      
        
    • Force anyone. robots.txt is a request, not a lock. Bots that ignore it need blocking at your firewall or CDN.
      • 
        
    • Remove content already collected. It only affects future visits.
      • 
        
    • Control Google AI Overviews or Bing Copilot. They use the normal Googlebot and Bingbot. Blocking those removes you from search too.
      • 
        
    • Guarantee AI tools cite you. Allowing bots only makes it possible.
      • 
        
    • Stay up to date by itself. New AI bots appear often; check this page from time to time.
      • 
      
    
    
    
  
    

    



    
  
    
    
    Cheat sheet
    Three kinds of AI bots
    Blocking one kind doesn't block the others.
    
    
    
      
    Training crawlers
    Collect pages to train AI models. Blocking them doesn't remove you from AI search.
    
        
    ExamplesGPTBot, ClaudeBot, Google-Extended, CCBot
    
      
    AI search crawlers
    Index pages so AI search tools can show and link to them. Block these and you can disappear from those answers.
    
        
    ExamplesOAI-SearchBot, Claude-SearchBot, PerplexityBot
    
      
    User-triggered fetchers
    Open a page because a person asked an AI about it. Some companies say these may not follow robots.txt, because a person asked.
    
        
    ExamplesChatGPT-User, Claude-User, Perplexity-User
    
    
    
  
    

    



    
  
    
    
    Step by step
    How to add it to your website
    
    
      
      
    1. Copy your current file
      Open yoursite.com/robots.txt, copy everything and paste it in the box above, so nothing is lost.
      2. 
      
    2. Make your choice
      Pick a quick choice or set each bot yourself, then copy the result.
      3. 
      
    3. WordPress with Yoast SEO
      Yoast SEO → Tools → File editor, paste and save. With Rank Math: Rank Math → General Settings → Edit robots.txt.
      4. 
      
    4. Any other website
      Upload the downloaded robots.txt to the top folder of your site, so it opens at yoursite.com/robots.txt.
      5. 
      
    5. Check for Cloudflare
      If you use Cloudflare's AI bot blocking or managed robots.txt, it can add rules of its own. Check the live file afterwards.
      6. 
      
    6. Test it
      Use the checker on this page with your website address and make sure each bot shows what you expect.
      7. 
    
    
  
    

    



    
  
    
    
    Questions
    Frequently asked questions
    
    
    
      
    Does blocking GPTBot remove my site from ChatGPT?
    Not from ChatGPT search. GPTBot collects pages for training; ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages that people ask about. Block only GPTBot if you want to stay findable in ChatGPT search but opt out of training.
    
      
    Will this stop my content from showing in Google AI Overviews?
    No. AI Overviews and AI Mode use the normal Googlebot. Google-Extended only controls use in Gemini training and grounding. To limit AI Overviews you would use nosnippet or max-snippet on your pages, which also affects normal search snippets.
    
      
    Do AI bots have to follow robots.txt?
    No. robots.txt is a public request. The major AI companies say their main crawlers follow it, but some bots have been reported to ignore it. For bots that ignore it, block them in your firewall, hosting panel or CDN such as Cloudflare.
    
      
    Will blocking AI bots hurt my Google ranking?
    Blocking the AI bots in this list does not change Googlebot, so normal Google Search ranking is not affected. Never block Googlebot or Bingbot unless you want to leave search.
    
      
    What happens to content AI companies already collected?
    robots.txt only affects future visits. It does not remove pages already collected or models already trained.
    
      
    How do I edit robots.txt in WordPress?
    WordPress makes a virtual robots.txt. With Yoast SEO use Tools, then File editor; with Rank Math use General Settings, then Edit robots.txt. You can also upload a robots.txt file to your site's top folder; a real file replaces the virtual one.
    
      
    Why paste my current robots.txt?
    So your existing rules, such as blocked admin folders and your sitemap, are kept. The tool replaces only the rules for the AI bots it knows and leaves everything else as it was.
    
      
    How does the website checker work?
    It reads the public robots.txt of the site you enter and applies the same matching rules crawlers use (RFC 9309): the most specific bot group wins, then the longest matching path. Nothing you enter is stored.
    
    
    
  
    

    



    
  
    
    
    Related tools
    What to do next
    
    
      
      
    1. Control AI botsYou are here
      2. 
      
    2. Guide AI to your best pagesLLMs.txt Generator
      3. 
      
    3. Check if AI can use your pageAI Visibility Checker
      4. 
      
    4. Write your meta tagsMeta Tag Generator
      5. 
      
    5. Add answer-ready FAQsFAQ Page Generator
      6. 
      
    6. More SEO toolsAll tools
      7. 
    
    
    
    See all Content Writing & SEO tools →