Free robots.txt generator with AI crawler controls
Make a correct robots.txt in a minute. Choose which search engines and AI crawlers can read your site.
All crawlers
The default for every bot not listed below.
- Googlebot · GoogleGoogle Search (also feeds AI Overviews).
- Bingbot · MicrosoftBing Search (also feeds Copilot and ChatGPT search).
AI crawlers
Pick a preset, then fine-tune each bot.
AI search
Lets AI answer engines find and cite your pages. Block it and you won't appear in their answers.
- OAI-SearchBot · OpenAIFinds pages to show and link in ChatGPT search.
- Claude-SearchBot · AnthropicIndexes pages for Claude's search answers.
- PerplexityBot · PerplexityIndexes pages for Perplexity answers.
- Meta-WebIndexer · MetaIndexes pages for Meta AI answers.
- DuckAssistBot · DuckDuckGoFinds pages for DuckDuckGo's AI answers.
User requests
Fetches a page only when a person asks the AI about it, like a browser.
- ChatGPT-User · OpenAIOpens a page when a ChatGPT user asks about it.
- Claude-User · AnthropicOpens a page when a Claude user asks about it.
- Gemini-Deep-Research · GoogleReads pages for Gemini Deep Research reports.
- Perplexity-User · PerplexityOpens a page when a Perplexity user asks about it.
- MistralAI-User · Mistral AIOpens a page when a Le Chat user asks about it.
AI training
Uses your pages to train AI models. Blocking it doesn't remove you from AI answers.
- GPTBot · OpenAICollects pages to train ChatGPT models.
- ClaudeBot · AnthropicCollects pages to train Claude models.
- Google-Extended · GoogleControls use of your pages for Gemini training (not Google Search).
- Applebot-Extended · AppleControls use of your pages for Apple Intelligence training.
- meta-externalagent · MetaCollects pages to train Meta AI (Llama).
- Amazonbot · AmazonCollects pages for Alexa and Amazon AI.
- Bytespider · ByteDanceCollects pages to train ByteDance (TikTok) models.
- CCBot · Common CrawlOpen web archive used by many AI companies for training.
- cohere-ai · CohereCollects pages for Cohere's AI models.
- Diffbot · DiffbotScrapes pages into data sold for AI and analytics.
# robots.txt for your website # Made with the free K2 Web robots.txt generator User-agent: * Allow: /
Upload it to the root of your site so it opens at example.com/robots.txt.
Allowedby Allow: / in the * (all bots) group
Make a robots.txt in three steps
- STEP 1
Add your site
Enter your website and sitemap, or load your current robots.txt to edit it.
- STEP 2
Choose the rules
Block private folders, then allow or block each AI crawler, or use a one-click preset.
- STEP 3
Test and upload
Test any page against any bot, then download robots.txt and upload it to your site's root.
What is robots.txt?
robots.txt is a small text file at the root of your website (example.com/robots.txt) that tells crawlers which pages they may visit. Search engines like Google and Bing follow it, and so do the main AI crawlers from OpenAI, Anthropic, Perplexity and Google. It controls crawling, not indexing: to keep a page out of search results, use a noindex tag instead.
Robots.txt generator FAQ
By using this tool you agree to its Terms of Use and privacy notice.
It depends on your goals. Blocking AI training bots (GPTBot, ClaudeBot, Google-Extended) keeps your content out of future model training. Blocking AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) also removes you from AI answers and their links, so most businesses allow those.
GPTBot collects pages to train OpenAI's models. OAI-SearchBot finds pages to show and link in ChatGPT search. You can block one and allow the other.
No. Google-Extended only controls whether Gemini can use your content for training. Googlebot, which powers Google Search, is separate.
At the root of your domain, so it opens at https://yoursite.com/robots.txt. In Webflow, paste it in Site settings → SEO → robots.txt. In WordPress, use your SEO plugin's robots.txt editor.
Not reliably. A blocked page can still appear in results if other sites link to it. Use a noindex meta tag (and allow crawling) to keep a page out of search.
Major search engines and AI companies say they do. Some scrapers ignore it, so never rely on robots.txt to protect private content.
Related free tools
People also search for
- robots.txt generator
- robots.txt for AI
- block GPTBot robots.txt
- block AI crawlers robots.txt
- robots.txt example
- robots.txt for Webflow
- robots.txt for WordPress
- disallow all robots.txt
- robots.txt tester
Questions about the robots.txt generator?
Stuck, found a bug, or have an idea to make it better? Send a message and we'll reply within 1–2 working days.
- Email us at pketu916@gmail.com
- Saw this tool used to deceive or harm someone? Choose Report misuse as the topic.
- Read the Terms of Use and Privacy notice for this tool.