Reading robots.txt, fetching the page without JavaScript, checking structured data…
One problem stops assistants from reading or quoting this page. Fix it first.
1,298 words in the HTML, loaded in 0.2 s, redirected to /.
| Crawler | Used for | Type | This page |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | AI search | Allowed |
| GPTBot | OpenAI model training | Training | Allowed |
| ChatGPT-User | ChatGPT, when a user asks | User request | n/a |
| Claude-SearchBot | Claude search | AI search | Allowed |
| ClaudeBot | Anthropic model training | Training | Allowed |
| Claude-User | Claude, when a user asks | User request | Allowed |
| PerplexityBot | Perplexity search | AI search | Allowed |
| Perplexity-User | Perplexity, when a user asks | User request | n/a |
| Googlebot | Google Search, AI Overviews and AI Mode | AI search | Allowed |
| Google-Extended | Gemini training (not Search) | Training | Allowed |
| Applebot-Extended | Apple model training | Training | Allowed |
| CCBot | Open dataset used by many models | Training | Allowed |
Blocking training crawlers is a choice and doesn't affect AI search. What each crawler does
No structured data found.
Fix: Add JSON-LD: Organization on the home page, Article with author and dates on articles, Product or SoftwareApplication on product pages.
How to fix it0 of 7 subheadings are phrased as questions.
Fix: Phrase some H2s as the questions people ask, with a one or two sentence answer right below each.
How to fix itNo machine-readable date or author found. Assistants use dates and authorship to judge whether a page is current and who stands behind it.
Fix: Show "Updated [date]" and the author, and add datePublished, dateModified and author to the Article JSON-LD.
How to fix itThe page has no canonical link.
Fix: Give every page a unique title under 60 characters, a description of 150 or so, and a canonical link to itself.
No /llms.txt. It is optional, and no major assistant has said it uses one, but it is cheap to add.
Fix: Add a short Markdown summary of the site with links to its key pages at /llms.txt.
How to fix itEvery AI search and answer crawler is allowed.
This check shows whether they can. Surfacepath Pro will ask them your customers' questions every week and show whether you're mentioned, cited, or replaced by a competitor.
1,298 words are in the HTML before any JavaScript runs.
How to fix itReturns HTTP 200 after 1 redirect with no noindex or nosnippet rules.
How to fix itrobots.txt points to a sitemap.