Skip to content

Free AI robots.txt generator: Decide which AI crawlers can read your website.

AI companies use their own crawlers, and robots.txt is where you allow or block them. This free generator writes the rules for the main AI user agents, including OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and Common Crawl's CCBot, in your browser.

Free AI robots.txt generator

Search engines (Google, Bing, Naver, Yandex, Apple…)
AI search & answer engines (ChatGPT search, Claude, Perplexity, Gemini agents, Meta AI…)
AI model training (GPTBot, ClaudeBot, Google-Extended, CCBot…)
SEO & comparison crawlers (Ahrefs, Semrush, Majestic…)
# Generated with the free robots.txt generator by Marketing Studio by UshbaTech: https://marketing.ushbatech.com/tools/ai-robots-txt-generator

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/

User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckDuckBot
User-agent: YandexBot
User-agent: Yeti
User-agent: Daum
User-agent: Baiduspider
User-agent: SeznamBot
User-agent: PetalBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Agent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: YouBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: cohere-training-data-crawler
User-agent: AI2Bot
User-agent: DeepSeekBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: DataForSeoBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-Signal: search=yes, ai-input=yes, ai-train=yes

Sitemap: https://example.com/sitemap.xml

Bot names are current as of November 2026. A named group does not inherit the rules of “User-agent: *”, so every group repeats your private paths.

What does this AI robots.txt generator do?

It lets you choose, crawler by crawler, whether AI training crawlers and AI search crawlers may access your site, then outputs robots.txt groups you can paste into your existing file. A common choice for businesses that want to be recommended by AI assistants is to allow search crawlers and decide separately about training crawlers.

How to create a robots.txt file for AI crawlers

  1. 1

    Step 1

    Choose your policy

    Allow everything, block AI training only, or pick crawler by crawler.

  2. 2

    Step 2

    Review the rules

    Check the generated User-agent and Disallow lines and keep your existing rules for search engines.

  3. 3

    Step 3

    Update robots.txt

    Paste the rules into the robots.txt at your domain root and test a few URLs.

Want this done for you?

Marketing Studio's AI employees create, publish and track your marketing across 8 social networks, email, ads and SEO, and its SEO & AI-SEO audit checks robots.txt, llms.txt and structured data automatically, with optional fixes committed to GitHub.

Start free with bonus credits

Free AI robots.txt generator: frequently asked questions

Still have questions? Talk to a human

Will blocking GPTBot remove me from ChatGPT answers?

Not necessarily. OpenAI uses GPTBot for model training and OAI-SearchBot for ChatGPT's search features, and blocking one does not block the other. If you want to appear in ChatGPT search results, keep OAI-SearchBot allowed, and decide separately whether GPTBot may train on your content.

#

What is Google-Extended?

Google-Extended is a robots.txt token that controls whether Google may use your content to train and ground its Gemini models. It is not a separate crawler, and blocking it does not affect your Google Search ranking or indexing; Googlebot rules continue to control search.

#

Do all crawlers obey robots.txt?

Reputable crawlers from search engines and major AI companies do, but robots.txt is a public request, not a lock, and bad bots can ignore it. Protect anything private with authentication, and use server rules or a firewall to block abusive crawlers that ignore your file.

#