What llms.txt does, and what it does not do

llms.txt is a Markdown file at the site root that tells an AI agent which pages are public and what they are for. It is useful as documentation. It is not a ranking lever, and Google's AI optimization guidance does not treat it as one.

Siteory reports llms.txt only when the crawl actually saw the URL. Absence is not scored as a failure.

What is llms.txt?

llms.txt is a convention, not a standard enforced by Google Search. The file usually lives at /llms.txt and contains:

• A one-sentence description of the product
• Links to the pages you are willing to have quoted
• Notes about limits, login walls, or preview data

A second file, llms-full.txt, is sometimes used for a longer dump. Most sites do not need it. If the public pages are already in HTML, a short map is enough.

Think of it as a polite README for agents, similar in spirit to robots.txt but written for people and models rather than for allow/disallow rules.

What llms.txt does not do

It does not:

• Make ChatGPT cite you
• Change Google rankings
• Replace robots.txt
• Unlock a hidden AI Overview feature
• Override a noindex tag or a login wall

If GPTBot is disallowed, a beautifully written llms.txt will not be fetched. If the homepage is a client-only shell, the file will describe pages the crawler cannot read.

Google has said that optimizing for generative AI search is still SEO. That means crawlable HTML, clear answers, and identity. llms.txt is optional completeness, not a substitute.

When you should add one

Add llms.txt when the public information architecture is easy to get wrong. A SaaS with a dashboard, a blog, and three feature pages is a good candidate. An agent that lands on /dashboard should not treat that route as documentation.

A good file:

• Lists only URLs that return 200 for anonymous users
• Uses the same titles as the pages themselves
• States product limits in plain language
• Avoids marketing claims you would not put on the homepage

Siteory's own file lists the audit, pricing, and how a report is used. It does not claim that llms.txt improves visibility. That honesty is the point.

How this relates to robots.txt

robots.txt remains the access control file. Allow or deny Googlebot, GPTBot, ClaudeBot, and PerplexityBot there. Keep private app routes disallowed. Keep the marketing site allowed.

llms.txt then explains the allowed surface. The two files should agree. If robots.txt blocks /blog and llms.txt promotes /blog, you have written a contradiction.

See crawlability, robots.txt, and sitemaps for the crawl side.

How Siteory treats the file in an AEO audit

The AI search audit records whether /llms.txt appeared in the discovered URLs. If it did, the report says so and asks you to confirm the file still describes the site. If it did not, the report says the file was not part of the crawl.

That is intentional. Turning a missing llms.txt into a red score would train teams to add an empty file for the badge. Empty files help no one.

If you want one, write four honest sentences and link the pages that already exist: features, blog, pricing, and about. Then spend the rest of the hour on the opening paragraph of those pages. That is the work AEO actually rewards.