What is a user agent?

In short

A user agent is the name a browser or crawler sends to identify itself when it requests a page, such as Googlebot or GPTBot. robots.txt rules are written per user agent.

Anyone can send a request claiming to be a known crawler, so the name alone isn't proof. OpenAI and Anthropic publish the IP ranges their crawlers use, so you can check that a visitor in your logs is genuine.

Updated

Guides that cover this

Related terms

robots.txtOAI-SearchBot
Check your own siteSee which AI crawlers your robots.txt lets in and what stops assistants quoting you.Run the free check