What is robots.txt?
robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may and may not fetch, using groups of rules per user agent.
A crawler follows the most specific group that names it, so a User-agent: GPTBot group overrides the * group for GPTBot. The format is standardized in RFC 9309.
robots.txt controls crawling, not indexing: a blocked page can still be listed by search engines if other sites link to it. To keep a page out of results, let it be crawled and use noindex.
Updated
Guides that cover this
How to allow GPTBot (and ChatGPT search) in robots.txt
2 min readAdd a User-agent: GPTBot group with Allow: / to robots.txt, and make sure no broader rule blocks it. If you want to appear in ChatGPT's search answers, allow OAI-SearchBot too. It's a separate crawler, and it's the one that decides whether ChatGPT search can show your pages.
Block AI training but stay in AI search answers
2 min readDisallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot allowed. The full file is below.