How the check works
One request for the page, one each for robots.txt, llms.txt and, if needed, sitemap.xml. Nothing runs in a browser, which is the point: that's how most AI crawlers see your site.
Scoring
Each item has a weight. A pass earns the full weight, a warning half, a failure nothing. The score is the share of the total. If your firewall turns our request away, that is reported as its own failure, and only robots.txt, the sitemap and llms.txt are scored.
| Check | Weight | How it is judged |
|---|---|---|
| AI crawlers can reach this page | 25 | robots.txt is read with a standards-compliant parser for each crawler below. Blocking two or more AI search crawlers, or Googlebot, fails; blocking one warns. Training crawlers are reported but not scored. |
| Content is readable without JavaScript | 20 | The page is fetched once without running scripts. Under 120 words with an empty app root or many scripts fails; under 250 words warns. |
| The page can be indexed and quoted | 15 | noindex in the robots meta tag or X-Robots-Tag header fails; nosnippet or max-snippet:0 warns. |
| Structured data says what the page is | 15 | JSON-LD and microdata types. Organization, Article, Product, SoftwareApplication and similar pass; only WebSite or WebPage warns; none fails. |
| Headings match the questions people ask | 10 | Checks for one H1, and warns when there are several subheadings but none phrased as a question. |
| Dates and an author are visible to machines | 5 | Looks for dates and authors in JSON-LD, meta tags and time elements. |
| Title, description and canonical | 5 | A missing title fails; a long title, no description or no canonical warns. |
| A sitemap lists your pages | 3 | A Sitemap line in robots.txt, or /sitemap.xml. |
| llms.txt | 2 | Present or not. Kept low because no major assistant has said it uses the file. |
Crawlers we read robots.txt for
OAI-SearchBot(OpenAI): ChatGPT searchGPTBot(OpenAI): OpenAI model trainingChatGPT-User(OpenAI): ChatGPT, when a user asksClaude-SearchBot(Anthropic): Claude searchClaudeBot(Anthropic): Anthropic model trainingClaude-User(Anthropic): Claude, when a user asksPerplexityBot(Perplexity): Perplexity searchPerplexity-User(Perplexity): Perplexity, when a user asksGooglebot(Google): Google Search, AI Overviews and AI ModeGoogle-Extended(Google): Gemini training (not Search)Applebot-Extended(Apple): Apple model trainingCCBot(Common Crawl): Open dataset used by many models
What it can't tell you
A good score means nothing technical is stopping assistants. It doesn't mean they will cite you: that depends on whether your page answers the question better than other sources. It also can't see firewall rules that treat real AI crawlers differently from us. Check your CDN's bot settings and server logs to be sure.
Our crawler
Requests come from the user agent SurfacepathCheck/1.0, only when someone runs a check, and are limited per visitor. We don't store pages. Questions: use the contact form. Run a check.