How to allow GPTBot (and ChatGPT search) in robots.txt

The exact robots.txt lines to let OpenAI's crawlers in, the difference between GPTBot and OAI-SearchBot, and how to set it up in Next.js.

Updated 2 min readBy Satchit
Short answer

Add a User-agent: GPTBot group with Allow: / to robots.txt, and make sure no broader rule blocks it. If you want to appear in ChatGPT's search answers, allow OAI-SearchBot too. It's a separate crawler, and it's the one that decides whether ChatGPT search can show your pages.

GPTBot, OAI-SearchBot and ChatGPT-User#

OpenAI runs three relevant user agents, and each is controlled separately:

  • GPTBot collects content that may be used to train OpenAI's models.
  • OAI-SearchBot finds pages for ChatGPT's search features. Block it and your pages won't appear in ChatGPT search answers, though they can still show up as plain navigational links.
  • ChatGPT-User opens a page when someone asks ChatGPT about it. Because a person triggers it, OpenAI says robots.txt rules may not apply.
Test your own siteSee which AI crawlers your site lets in and what else stops them quoting you.

The robots.txt rules#

To let both crawlers in while keeping a private area closed:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

A crawler follows the most specific group that names it, so a User-agent: GPTBot group overrides the * group for GPTBot. That also works the other way: if an old User-agent: GPTBot / Disallow: / block is still in the file, it wins over anything you add to *. Search the file for every mention of the bot before you assume it's allowed.

In Next.js#

With the App Router, create app/robots.ts and Next.js serves it at /robots.txt:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: ['GPTBot', 'OAI-SearchBot'], allow: '/' },
      { userAgent: '*', disallow: '/admin/' },
    ],
    sitemap: 'https://yoursite.com/sitemap.xml',
  }
}

Check that it worked#

Open yoursite.com/robots.txt in a browser and read the result: generated files sometimes differ from what you expect. Then run the free check, which reads your robots.txt the way each crawler does and shows which ones are allowed for the page you enter.

Source: OpenAI's crawler documentation.

Questions

Does allowing GPTBot help my Google rankings?

No. Google uses its own crawlers. GPTBot only affects OpenAI.

Can I allow ChatGPT search but block training?

Yes. Allow OAI-SearchBot and disallow GPTBot. OpenAI reads the two separately.

How long does a robots.txt change take to apply?

OpenAI says changes can take about 24 hours to show up in ChatGPT search results.

Related guides