If the goal is to climb positions in Google or increase the chance of being cited by AI answers, llms.txt should not be treated as a proven shortcut. As of August 2026, available evidence points to a more specific function: offering agents and tools a lean, machine-readable map of content that is already public and useful. The gain may exist in documentation and agentic flows, but there is no basis for selling the file as a ranking factor or visibility button in AI.
This distinction became clearer throughout 2026. On June 15, Google Search Central noted that Google Search does not use llms.txt and that maintaining the file neither improves nor harms visibility or rankings. In parallel, Chrome started to verify it in an experimental Lighthouse category focused on agentic navigation. And, on August 10, the format proposal reached version 2, with changes to facilitate the discovery of the file and Markdown versions of the pages. The same name, therefore, appears in different contexts: search, documentation and agents are not the same thing.
What llms.txt was created to do
Jeremy Howard's original proposal, published in September 2024, starts from a practical problem: web pages are made for people and can carry navigation, scripts, ads and a lot of peripheral content. An agent that just needs to understand an API, a policy, or the structure of a website may spend too much context and time finding what matters.
llms.txt attempts to resolve this with a Markdown file that introduces the project or website and points to relevant resources. Version 2 allows it to exist in the root, like /llms.txt, or in a specific path, like /docs/llms.txt. In this second case, the file can serve as a map of just that area of the site. The proposal also recommends clean Markdown versions of pages when this makes sense.
The central point is curation. A sitemap tends to represent URLs that the site wants to make discoverable; llms.txt is intended to highlight a smaller set of materials that help an agent.
guide yourself and seek details when necessary. It is not a permission file and does not replace tracking controls.
What changed in the proposal in August 2026
Version 2, modified on August 10, 2026, added an answer to a problem that the first version left open: how does an agent discover llms.txt without simply trying to guess its existence? The new proposal recommends standardized link relationships. The describedby relationship can point to the llms.txt applicable to the page, while alternate with the text/markdown type can indicate the Markdown version of that content.
The revision also formalized the use of subdirectories and began to accept more than one form of URL for Markdown versions. The logic became less like a magic file that every agent should look for at the root and more like a navigation layer that can be explicitly discovered by compatible tools.
Still, llms.txt remains an open proposal, not a mandatory web standard. Adoption by documentation platforms and tools shows real interest, but does not, in itself, create an obligation for support by search engines or wizards.
llms.txt, robots.txt and sitemap.xml do different jobs
robots.txt expresses tracking rules
Robots.txt is part of the Robots Exclusion Protocol, standardized in RFC 9309. It informs crawlers which paths can or cannot be accessed according to the user-agent. It's a policy for cooperative robots, not a security or authentication barrier. If an area is private, it needs its own access controls.
This difference matters because OpenAI and Anthropic, for example, publish specific guidance about their crawlers and how site owners can allow or restrict access via robots.txt. To appear in ChatGPT results and summaries, OpenAI's official guidance highlights the importance of not blocking OAI-SearchBot. Anthropic also documents distinct bots for training, searching, and user-initiated access, all with preferences controlled by robots.txt.
sitemap.xml helps engines discover URLs
The sitemap tells search engines which pages, videos and other files the site considers relevant, and can include metadata. Google uses it to crawl sites more efficiently, although it is clear that placing a URL in the sitemap does not guarantee crawling or indexing.
For an editorial or institutional website, this function usually has a more direct operational impact than creating an llms.txt: an updated sitemap, crawlable internal links and accessible pages help engines find content. llms.txt does not replace any of these tasks.
llms.txt provides context and curated paths
The llms.txt tries to be an index for consumption by agents. Instead of controlling access or listing everything, it describes what the site is and points to selected materials. Its usefulness depends on there being a consumer who knows how to find it, read it and use the links. Without this chain, the file could be technically correct and still never influence a single answer.
Who actually reads llms.txt in 2026
The response needs to separate three things: publishing the file, making a request to the file and using the contents of the file to decide a response. These steps are often mixed.
OpenAI's own API documentation now offers an llms.txt with links to Markdown guides and references. The Anthropic documentation also provides such a file, and the Perplexity documentation does the same. This shows that the format has value as documentation packaging for consumption by tools. It does not prove that these companies' search products automatically use any llms.txt found on the web.
The most explicit case of official consumption appears in the Gemini API ecosystem. In July 2026, Google's documentation began recommending an MCP server to keep programming assistants updated and informs that certain skills can search llms.txt in ai.google.dev as an alternative when MCP is not installed. Here there is a concretely described consumer: a programming agent configuration that uses the file as a source of documentation.
Chrome provides another example of a real request, but with a different purpose. Lighthouse's experimental Agentic Browsing category checks for the presence of llms.txt. If the file does not exist and returns 404, the audit is not applicable; If a server error occurs when trying to fetch it, the page may be flagged. This proves that the tool is trying to recover the file, not that Google Search uses it for ranking.
As for Google Search, the official answer is straightforward: the file is ignored for visibility and classification, including in the search's generative AI features. This resolves a common confusion created by different teams at Google working with llms.txt in agent documentation or testing.
What the logs show when we leave theory
One of the largest public samples to date was published by Ahrefs on June 15, 2026. The company analyzed 137,210 domains that used its Web Analytics and had received traffic in May. About 28% published a valid llms.txt, but 97% of these files did not receive any requests during the month.
Among the files that received traffic, 96% of requests came from bots. Ahrefs classified 19.5% of requests as coming from identified AI tools and found GPTBot and Claude-Code among the most frequent readers of this subset. The company also noted that much of the traffic came from tools that studied or audited the llms.txt ecosystem itself.
These numbers should not be transformed into a universal truth. Ahrefs itself recognizes that its base tends to bring together sites that are more technical and SEO-conscious than the web in general. The cutoff is also one month. And an HTTP request only proves that a client fetched the file; does not prove that the content was incorporated into context, improved an answer, or generated a citation.
Even with these limitations, the study is useful because it overturns an assumption: publishing llms.txt does not mean that agents will show up to read it. On most sites observed, this simply did not happen.
Searching the archive is not the same as improving citations
This is the most important separation for any business decision. A bot might ask for /llms.txt for auditing, training, curiosity, generic indexing, or a tool's routine. None of this demonstrates that the file increases the likelihood of your brand being cited in a response to a user.
To claim visibility gains, it would be necessary to link the implementation to a measurable result: more appearances, more citations, better response recovery or more qualified traffic, controlling other changes on the website. Public evidence from 2026 does not generally support this relationship.
So the correct question is not just who accessed the file, but what happened afterwards. Server logs can show user-agent, time, path, and HTTP status. Analytics tools can show references coming from AI services. Response tracking can track citations. Still, attributing causality to llms.txt requires caution, because content, links, authority, tracking, and the AI product itself change at the same time.
Decision matrix by type of site
Editorial blog or institutional website
For a typical blog, content portal or institutional website, llms.txt is low priority. If the infrastructure generates the file automatically, with almost zero maintenance and without exposing unwanted URLs, maintaining it may be acceptable technical hygiene. But it shouldn't take the place of original content, good information architecture, internal links, sitemap, performance, semantic HTML and correct tracking of the bots you actually want to receive.
If the main motivation is to appear on Google or in Google Search AI resources, Google itself has already eliminated the doubt: llms.txt neither helps nor hinders. For ChatGPT Search and Claude, the official guidance available to publishers places more weight on content accessibility and crawler rules than on this file.
Technical documentation and APIs
Here the scenario changes. SDK, API, library, or technical product documentation is the most compelling use case. Programming agents need to quickly find references, examples, versions, and guides. A table of contents curated in Markdown can reduce noise and point to pages in a template-friendly format.
It is precisely in this environment that we see concrete adoption: documentation platforms generate llms.txt, AI labs publish it on their own technical portals, and the Gemini API ecosystem describes a skill that can fetch it. For large and changing documentation, implementing the file may have operational utility even without any promise of SEO.
SaaS Product
In a SaaS, the decision depends on where the informational value lies. If your product has public documentation, API, extensive help center, technical changelog, or integration guides, it makes sense to consider an llms.txt specific to that area, rather than turning the entire domain root into a catalog for agents.
If the SaaS only has commercial pages, prices and landing pages, it is closer to an institutional website. In this case, the archive may be cheap to maintain, but it is unlikely to overcome basic discoverability, content, and crawling priorities.
What to prioritize before creating llms.txt
Ensure the right bots can access content
Check robots.txt, firewall, CDN, bot protection, and HTTP responses. OpenAI directs publishers interested in ChatGPT Search to enable OAI-SearchBot, and Anthropic documents its own agents for user-initiated search and access. A perfect llms.txt file cannot compensate for important pages blocked by 403, improper authentication, or conflicting crawler rules.
Make human architecture work first
Crawlable internal links, stable URLs, updated sitemap, content present in the DOM, semantic HTML and coherent navigation remain basic infrastructure. Google's guide to generative AI capabilities reinforces that good SEO practices and useful content remain the foundation, without the need for special AI files.
Give agents content that’s actually worth retrieving
An index does not improve weak material. Documentation needs to be correct, updated and clear; commercial pages need to answer real questions; Editorial content needs to add original information. If llms.txt just points to generic or outdated pages, it creates a problem rather than solving it.
How to implement without turning the file into a marketing promise
Keep the scope small and curated
The current proposal only requires a project or website title and allows you to add summary, context and sections with described links. Instead of copying the entire sitemap, select the features that an agent would actually need to understand that area. For documentation, prefer guides, API references, relevant policies, changelog, and canonical pages.
Use specific paths when it makes sense
Version 2 formalized llms.txt into subdirectories. A SaaS might, for example, maintain the file only in documentation. This reduces noise and makes the scope clear. It also avoids mixing marketing, support, documentation pages and areas with completely different objectives.
Make discovery explicit if your stack allows it
The v2 proposal recommends link relationships to point from the page to the applicable llms.txt and corresponding Markdown version. This is more robust than expecting every agent to try /llms.txt by convention. Still, the usefulness will depend on the support of the tool that reaches the site.
Measure requests and treat usage as a hypothesis
After publishing, monitor the logs. Record which user-agents request the file, how often, which paths they follow, and whether there are associated referral or traffic patterns. Do not interpret a simple request as proof of improved ranking or citation. The objective of measurement is to discover whether there is a real consumer in your context.
So, is it worth implementing?
In 2026, the best answer is conditional. For blogs and institutional sites, llms.txt is generally a low priority task and, when done, should be viewed as cheap hygiene, not an SEO investment. For technical documentation, APIs, and products used by developer agents, there is a much stronger practical case because the format organizes content that these tools actually need to query.
For SaaS, the decision is usually in the middle: it is worth more when there is rich public documentation and less when the site is essentially commercial. In any scenario, do the basics that already have proven consumers first: useful content, crawling allowed, sitemap, internal links, accessible HTML, and up-to-date documentation.
llms.txt can be a good additional layer. What it is not, at least on the evidence available in August 2026, is a guarantee of discovery, ranking or citation by AI systems.
