Block AI training but stay in AI search answers

A robots.txt that opts out of model training while keeping your pages available to ChatGPT search, Claude, Perplexity and Google AI Overviews.

Updated 2 min readBy Satchit
Short answer

Disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot allowed. The full file is below.

Opting out of training and staying visible in AI answers aren't in conflict. The companies use different crawler names for the two jobs, so you can say no to one and yes to the other.

The robots.txt#

# Training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# AI search and answers: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Everyone else, including Googlebot
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml
Test your own siteSee which AI crawlers your site lets in and what else stops them quoting you.

What you give up#

Very little that's measurable today. Training crawlers don't send visitors. The trade-off is long term and uncertain: a model that never saw your content may know less about your brand when it answers without searching. If being known by name matters more to you than controlling reuse, leave training allowed.

In Next.js#

import type { MetadataRoute } from 'next'

const training = ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot']
const search = ['OAI-SearchBot', 'Claude-SearchBot', 'Claude-User', 'PerplexityBot']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: training, disallow: '/' },
      { userAgent: search, allow: '/' },
      { userAgent: '*', allow: '/' },
    ],
    sitemap: 'https://yoursite.com/sitemap.xml',
  }
}

After deploying, run the checker on a page. Its crawler table should show the training bots blocked and every search crawler allowed.

Questions

Will blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search. That uses OAI-SearchBot, which you can keep allowed. Blocking GPTBot tells OpenAI not to use your content for training.

Does this remove content that was already collected?

No. Robots.txt applies to future crawling. Anthropic, for example, says blocking ClaudeBot excludes your future content from training.

Is Google-Extended needed to stay out of Gemini training?

Yes. Google-Extended is the token Google provides for that. It doesn't affect Google Search or AI Overviews.

Related guides