robots.txt leaking private routes in AI-built apps

A Disallow entry is a roadmap, not a lock

robots.txt is public by design. It lives at the root of your site, at /robots.txt, and anyone can read it with a single GET request and no login. It is often the first file an attacker or a scanner fetches, because it is small, it is always at the same path, and it tells them what you were thinking when you built the site.

The concrete risk is that a Disallow line does not protect a path. It points at one. A robots.txt that contains Disallow: /admin, Disallow: /api, Disallow: /internal, Disallow: /dashboard, or Disallow: /staging is telling every reader exactly which paths you consider sensitive, while doing nothing to stop a request to them. robots.txt is a request to crawlers, honored only by polite ones. It is not enforced by your server, it is not checked on incoming requests, and it has no effect on a human typing the path into a browser or a script that ignores the file entirely.

What is observable from outside is exact: the literal contents of your robots.txt and any sitemap it links. What that reveals is a list of the paths you wanted hidden. Whether those paths are actually protected by a login is a separate question, and the answer is often no, which is the whole problem.

• A Disallow line names a path you treat as private and broadcasts it to anyone who reads the file.
• robots.txt is advisory. Well-behaved crawlers like Googlebot respect it; attackers, scanners, and curious visitors are under no obligation to.
• This is information disclosure (CWE-200 territory), not a broken login. The file is doing its documented job; the mistake is expecting it to do a different one.

Why AI-built apps generate over-eager Disallow lists

robots.txt has legitimate uses. Keeping search engines out of duplicate pages, faceted-search URLs, or a print view is normal and useful. The failure is not having a robots.txt; it is treating it as a security boundary and stuffing it with the names of paths you actually want kept private.

AI coding tools and project templates tend to generate a thorough-looking robots.txt because thorough looks correct. Asked to set up SEO or a robots file, a tool will often produce a long Disallow list covering every path that sounds administrative or internal: /admin, /api, /login, /dashboard, /internal, /staging, /backup. It reads like a security measure, and it ships in one pass without a review step that asks whether naming those paths in a public file is a good idea.

• The generated file looks complete and professional, so it rarely gets a second look.
• The tool optimizes for not letting crawlers near sensitive areas, which is a reasonable instinct applied to the wrong tool. The result is a public index of your sensitive areas.
• A linked sitemap can make it worse. An auto-generated sitemap.xml sometimes lists private or staging URLs directly, so even a clean robots.txt can leak paths through the sitemap it references.

Read your own robots.txt and sitemap from the outside

You can confirm exactly what you are revealing with no attack and no credentials. Fetch your own robots.txt and read it as an outsider would. Do this against your own URL.

Request /robots.txt on your public URL, from a logged-out session, and read every line. Then follow any Sitemap: line it contains, fetch that sitemap, and read the URLs it lists. A sitemap can link other sitemaps, so check those too.

The point of the check is to read the file as a stranger, not as the author. You know what /admin is for; the value here is seeing that you handed that knowledge to everyone who fetches one public file.

• Suspicious result: a Disallow entry that names an administrative, internal, or API path, for example /admin, /api, /internal, /dashboard, /staging, /backup, /private, or a path containing a token or an unguessable slug you intended to keep secret. Treat each such line as a path an attacker now knows to probe.
• Safer result: a generic robots.txt that controls crawling of public pages (Disallow on duplicate or low-value URLs) and does not enumerate sensitive areas, paired with a sitemap that lists only pages you are happy to have indexed.
• For each sensitive path the file reveals, go and request that path from a logged-out session. If it returns the page or its data without asking you to log in, the robots.txt did not just advertise the path, it advertised an unprotected one. That second request is the one that turns a disclosure into a finding.

Use robots.txt for crawl control, and protect paths server-side

The fix separates two jobs that robots.txt is often asked to do at once: stop indexing, and stop access. robots.txt does neither reliably, and it was never meant to do the second.

First, keep robots.txt generic. Use it to steer crawlers away from genuinely low-value public URLs, and remove every Disallow line that names a sensitive path. Do not list /admin, /api, /internal, or any secret path. If a path should be private, leaving it out of robots.txt is strictly better than naming it, because a path you do not mention is one fewer path you have advertised.

Second, protect sensitive paths with authentication and authorization on the server. This is the only thing that actually stops access. An admin route is safe because the server checks that the caller is a logged-in admin on every request, not because a public file asked crawlers to stay away. Authentication confirms who the caller is; authorization confirms they are allowed to reach that specific path. robots.txt does neither.

Third, separate stop-indexing from stop-access. If your goal is only to keep a public page out of search results, the tool for that is a noindex directive (a meta robots tag or an X-Robots-Tag response header) on the page itself, which a crawler reads after fetching the page. If your goal is to stop access, only authentication does that. A page can be reachable by anyone and still be kept out of Google with noindex; a page can be in Google and still be locked behind a login. They are different controls for different problems.

• Crawl control (robots.txt): a request to polite crawlers about what to fetch. Public, advisory, no access enforcement.
• Index control (noindex): keeps a fetched page out of search results. Still publicly accessible to anyone with the URL.
• Access control (authentication and authorization, server-side): the only one of the three that decides whether a caller may reach the path at all.

Where VibeCodeGuard fits, and what it does not prove

VibeCodeGuard's Launch Check fetches your /robots.txt from the public surface and flags Disallow entries that point at administrative or API paths, the lines that hand an attacker a list of where to look. It reports those entries with severity and fix direction, so an over-eager Disallow list shows up in your own scan rather than in someone else's reconnaissance.

What the robots.txt signal does not do is confirm whether those paths require a login. That is a separate check. The scan tells you where to look; whether /admin is actually protected is something you confirm by requesting it from a logged-out session, and by reviewing your own auth on those routes. A flagged Disallow line means you have disclosed a path, not necessarily that the path is open, though the two often travel together.