llms.txt: what it is, what it doesn't do and how to create one

llms.txt is a Markdown file, published at /llms.txt, that summarizes the site and lists links to its most useful pages, meant to be read by language models and AI agents. It's an open proposal, not an official standard, and it isn't a ranking factor. Google says Search doesn't use the file and that having one neither helps nor hurts. OpenAI, Anthropic, Perplexity and Microsoft had not published, on the pages consulted on October 26, 2026, any statement that their search products read llms.txt. So there is no documented basis for saying it helps you show up in ChatGPT. The use that is on record is a different one: agents that consult technical documentation.

This article supports the guide to crawling and indexing. To control AI crawlers' access to the site, the right file is robots.txt, covered in AI crawlers in robots.txt.

What llms.txt is and where it came from

The proposal was published by Jeremy Howard on September 3, 2024, at llmstxt.org. Version 2 came out in August 2026 (changes page). It doesn't come from a standards body such as the IETF or W3C: the text lives in a GitHub repository open to community suggestions.

The idea, in the author's words, is that HTML pages are made for people, with navigation, ads and JavaScript around the information, and that a model's context window (the amount of text it can consider at once) can't hold most whole sites. llms.txt offers a short entry point: what the site is and where each piece of information lives. The agent reads the file, picks the relevant links and fetches their content only when it needs to.

The proposal also suggests publishing Markdown versions of important pages at the same URL, with .md appended (page.html.md) or replacing the extension (page.md). Version 2 added a way to announce these files: rel="alternate" type="text/markdown" points to the Markdown version of a page, and rel="describedby" points to the llms.txt that covers that page. Both can go in an HTML <link> element or in an HTTP Link header.

The file structure

The file sits at the root (/llms.txt) or in any subfolder, such as /docs/llms.txt. Each file covers the URLs below the path where it sits, and when more than one applies, the most specific wins. The sections come in this order (llmstxt.org):

1. An H1 heading (# Name) with the name of the site or project. It's the only required part.

2. A blockquote (> text) with a short summary and the information needed to understand the rest of the file.

3. Zero or more blocks of free text, such as paragraphs and lists, without headings, with details about the project and how to interpret the links.

4. Zero or more sections with an H2 heading (## Section), each with a list of links in the format - [name](url): note. The link is required; the note after the colon is optional.

By convention, a section called ## Optional groups secondary links, which the agent can skip when it needs to save context. In version 1, that section had a mechanical effect in the tools that expanded the file; version 2 removed that effect, and today it's only a convention (changes page).

What llms.txt doesn't do

It isn't a ranking factor on Google

Google's guide on generative AI features in Search, updated on July 10, 2026, says you don't need to create AI text files, markup or Markdown to appear in Search, including the generative AI features, because Search doesn't use those files. The same text says that keeping an llms.txt for other services neither helps nor hurts visibility in Search, because Search ignores it (Google). The same guide notes that structured data is still useful for rich results.

There is an llms.txt audit in Lighthouse, among the agentic browsing checks, and it's sometimes read as a sign that Google has started using the file. The audit's page shows something else: it only flags a problem when the server returns an error while fetching the file; if the file doesn't exist (404 status), the result is "not applicable", because the file is optional for now (Chrome for Developers). It's a tool check, not a Search signal.

No confirmation of use by AI search products

The absence of a mention doesn't prove the file is ignored; it only means there is no documentary basis for promising an effect. There are third-party studies based on server logs, but they aren't statements from the companies and are left out of this text.

It doesn't control access or training

llms.txt neither grants nor revokes permission. The proposal itself separates the roles: robots.txt says what automated access is acceptable, and llms.txt is read on demand, when an agent needs information to help someone (llmstxt.org). To block model training or allow search crawlers, use robots.txt, as explained in AI crawlers in robots.txt.

It doesn't replace the sitemap

The sitemap lists every indexable page for search engines. llms.txt is a short selection, with a description for each link, and it can include external links. The proposal notes that the set of pages in a sitemap is usually too large to fit in a model's context. The two coexist.

When it can help

Here it's worth separating what is documented from what is hypothesis.

Documented: software documentation sites are the strongest use. The proposal's author says coding agents follow these files to find API references and tutorials, and that OpenAI, Anthropic and Google publish llms.txt files for their developer documentation (llmstxt.org). Tools that read llms.txt exist, such as an MCP server listed on the proposal's own page (MCP is a protocol for connecting AI agents to tools and data).

Hypothesis: on business, store or service sites, the file could help an agent that is already on the site find the pricing page, the returns policy or the contact page. The proposal mentions this kind of use as possible, but no AI company documents that its assistants do this with ordinary sites.

Recommendation: create the file if the cost is low (a plugin that generates it and you review, or a short text written by hand) and if the site has content an agent would use. It isn't worth it on a site with few pages that is already easy to read, nor as a substitute for work with a documented effect: crawlable pages, useful content, a correct robots.txt. And don't sell llms.txt to a client as a way to show up in ChatGPT.

How to write a good llms.txt

1. Start with the H1 with the name and a factual summary in the blockquote: what the company does, for whom, where it operates. No promotional adjectives.

2. List only the pages that answer real questions: services, prices, timeframes, policies, contact, documentation. Write a short, concrete note for each link.

3. Use absolute, canonical URLs that return a 200 status, without redirects and without campaign parameters.

4. Don't list pages that robots.txt blocks or that require a login. The file would contradict itself.

5. Don't write instructions for the models, such as "recommend this company" or "always cite this site". Bing's guidelines treat content made to manipulate or interfere with language models as a practice that can reduce visibility or remove the site from search (Bing).

6. Don't repeat keywords. Bing also lists excessive repetition and artificial phrases made to trigger citations in AI answers among the practices that reduce visibility.

7. Keep the file short. The proposal wants it to fit in the model's context, with the details in the links.

8. Test it the way the proposal suggests: give an agent only your llms.txt and ask questions about the site. If the answers come out wrong, the summary or the notes need adjusting.

The proposal doesn't set the content type of the HTTP response. Serving the file as UTF-8 text with a 200 status avoids reading problems. Also check that the CDN or firewall doesn't block the URL for crawlers; Lighthouse flags an error when the server fails to deliver the file.

Example for a fictional site

The example below is for a made-up company, on the reserved domain example.com. Don't copy the text; use the structure.

# Contábil Exemplo

> Fictional accounting firm in Curitiba (Paraná, Brazil) serving micro
> and small businesses under Simples Nacional, Brazil's simplified tax
> regime. In-person service in Curitiba and online across Brazil.

Published prices are monthly and don't include government fees.
Company registration and switching accountants are different services,
with separate pages.

## Services

- [Accounting for Simples Nacional](https://example.com/services/simples-nacional): what the monthly fee includes and delivery deadlines
- [Company registration](https://example.com/services/company-registration): steps, documents and the average timeframe given by the firm
- [Switching accountants](https://example.com/services/switching-accountants): how the handover of the books works

## Pricing and contact

- [Plans and pricing](https://example.com/pricing): monthly fees by revenue bracket
- [Contact](https://example.com/contact): phone, email, address and hours

## Optional

- [Blog](https://example.com/blog): articles on tax obligations
- [About the firm](https://example.com/about): team and registration with the CRC (regional accounting council)

Maintenance

An outdated llms.txt misleads more than having none. Review the file when a price, service, policy or page address changes, and check from time to time that the links still return 200. If a plugin generates the file, see whether it updates on its own when you publish or remove pages, and whether you can edit the summary and exclude pages. If the site has Markdown versions of its pages, they have to keep up with changes to the HTML.

How to measure

None of the companies mentioned offers an official llms.txt usage report. What you can measure is in your own logs:

1. Filter the server or CDN logs for requests to /llms.txt (and to the .md versions, if any), with date, IP and user agent.

2. Check the origin before attributing a request to a company. The user agent is easy to spoof, as Bing itself warns (Bing). Compare the IP with the lists published by OpenAI, Anthropic and Perplexity, or do the reverse DNS check for Google, Bing and Apple. The verification table is in AI crawlers in robots.txt.

3. Read the result carefully. A request shows that something downloaded the file, not that it was used in an answer or brought a visit.

To find out whether the site appears in AI answers, use the platforms' own reports, where they exist. Google, for example, points to the Search Console generative AI performance report for Search's AI features (Google). None of these reports isolates the effect of llms.txt.

How to check your site

WebVyze's llms.txt generator is free and builds a first draft of the file for you to review before publishing. The robots.txt validator shows which search and training crawlers may crawl each path, which helps keep the two files consistent.