Crawlability, robots.txt, and XML sitemaps

Crawlability is whether a bot can fetch a URL and is allowed to index it. It is the first chapter of every serious website audit because later chapters are wasted if the page is blocked.

Siteory records robots.txt, status codes, and sitemap URLs during the crawl. This is how to read those findings.

Why can Google not see the new site?

The usual causes are a leftover Disallow: /, a sitewide noindex from the staging template, or a sitemap that still points at a preview host. None of those are ranking problems. They are access problems.

Check in this order:

1. Fetch the URL unauthenticated. Note the status and the final host after redirects
2. Fetch /robots.txt from the same host
3. View source and search for noindex and rel=canonical
4. Fetch /sitemap.xml and open a few listed URLs
5. Confirm the CDN is not adding X-Robots-Tag: noindex on HTML

If any step fails, stop. Do not rewrite titles yet.

robots.txt rules that surprise teams

robots.txt is case-sensitive in spirit and easy to over-block. A rule that was meant to hide /admin can hide /administration-guide. A rule that blocks /api can be fine. A rule that blocks /_next/static can break rendering for Googlebot even when the HTML is allowed.

Allow the marketing site. Disallow dashboards, settings, and login. Be explicit about AI crawlers only if you have a reason to treat them differently. Most public docs should be allowed for Googlebot, GPTBot, ClaudeBot, and PerplexityBot.

Sitemap declarations belong in robots.txt. The line should use the production origin, not localhost.

A preview environment should disallow everything. Production should not copy that file.

noindex, canonicals, and the staging leftover

noindex can arrive as a meta tag or as X-Robots-Tag. Either one is enough. A launch checklist that only greps the homepage HTML will miss a header set at the edge.

Canonical tags should be absolute and should point at the URL you want indexed. Common failures:

• Canonical to http on an https site
• Canonical to the www host when the site redirects to the apex, or the reverse
• Canonical to a parameterized URL
• Canonical copied from a component so every blog post points at /blog

Self-referencing canonicals on the preferred URL are the boring, correct pattern.

XML sitemaps that help, and ones that lie

A sitemap is a hint. It should list indexable 200 URLs you care about, with lastmod dates you actually update. It should not list /login, /dashboard, or thank-you pages.

Split only if you must. A marketing site with two dozen URLs needs one sitemap. Include the homepage, pricing, feature pages, legal pages, the blog index, and every article.

Lastmod should change when the content changes, not on every deploy. A sitemap that stamps today on every URL teaches crawlers to ignore the date.

Siteory's own sitemap is generated from the public route list so a new article cannot be forgotten.

JavaScript rendering is not a crawl strategy

Googlebot can render. Many other crawlers, including several AI bots, fetch the first HTML response and stop. If the article body is empty until a client bundle runs, those bots index a shell.

Server-render the copy you want quoted. Keep the interactive bits as islands. That is also the AEO requirement, not a separate project.

The crawl pass in Siteory

The SEO audit fetches the submitted URL, follows the same-host links it is allowed to follow, and records robots, status, canonical, and sitemap membership.

Fix the red crawl items before you buy another tool. Then run the AI search and security passes on the same host so the Markdown report describes one website, not three disconnected scores.