SEO · 4 min read
robots.txt for GPTBot, ClaudeBot, and Google-Extended
A GEO program that blocks the crawler for the engine it wants to rank in is a contradiction.
Published
robots.txt is still the access-control file. AI crawlers did not replace it. They added user-agents you have to name on purpose.
Split Search, fetching, and training
- Googlebot (and related Google crawlers) — Search, including AI Overviews and AI Mode when those features use the index.
- Google-Extended — Google’s control for using content to train Gemini. It is not the Search crawler. Blocking Google-Extended does not remove you from Search.
- GPTBot — OpenAI’s crawler for ChatGPT browsing / fetching, depending on current OpenAI docs.
- ClaudeBot / Applebot / PerplexityBot — same idea: named agents with their own documentation.
Read the vendor’s current robots page before you copy a gist from 2024. Agents get renamed.
A policy that does not fight itself
If you want to be cited in ChatGPT, do not Disallow: / for GPTBot on the public marketing site. If you want to keep the app private, disallow the app origin and allow the docs origin.
Example shape, not a paste-without-thinking snippet:
User-agent: GPTBot
Allow: /blog
Allow: /features
Disallow: /dashboard
Disallow: /login
Keep the same spirit for other AI bots. Keep Googlebot allowed on public pages if you want Search at all.
What Siteory will tell you
A crawlability audit should record whether robots.txt exists, parses, and contradicts the sitemap. An AI search audit should not invent a ranking penalty for a blocked bot. It should say: this engine cannot fetch the page you hoped it would cite.
llms.txt does not override robots.txt. If robots blocks the blog and llms.txt promotes the blog, you wrote a contradiction. See what llms.txt does.