In short
llms.txt is a plain markdown file at the root of a website that lists its most important pages so an AI model can find them quickly. It was proposed in September 2024. No major AI company has confirmed using it for ranking or citation, so treat it as cheap, useful documentation rather than an SEO tactic.
Key points
- It is a markdown file at yoursite.com/llms.txt listing your key pages with one-line descriptions.
- It was proposed by Jeremy Howard of Answer.AI in September 2024, not by any AI company.
- No major assistant has publicly confirmed it affects citation or ranking.
- It takes about twenty minutes to write and cannot hurt you.
- Do it for completeness and for agents pointed at it directly. Do not expect traffic from it.
The short version
llms.txt is a plain markdown file at the root of your site that lists your most important pages so an AI model can find them quickly. It was proposed in September 2024. No major AI company has confirmed using it for ranking or citation, so treat it as cheap, useful documentation rather than an SEO tactic.
Most of what has been written about llms.txt oversells it. This page will not take long, because the honest answer does not need long.
Where it came from
llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024. It is a community proposal, not a standard published by OpenAI, Anthropic, Google or Perplexity, and not something any of them asked for.
That distinction matters. robots.txt works because crawler operators agreed to obey it. sitemap.xml works because search engines agreed to read it. llms.txt has no such agreement behind it yet. It is a good idea that is waiting to be adopted.
The three files, and what each actually does
| File | Answers the question | Honoured by |
|---|---|---|
robots.txt | "Are you allowed in?" | All major crawlers |
sitemap.xml | "What pages exist?" | Search engines |
llms.txt | "Which pages actually matter?" | No confirmed consumer yet |
The common mistake is treating llms.txt as a replacement for either of the others. It is not. If you want to control AI crawler access, that is robots.txt. If you want your pages discovered, that is sitemap.xml and internal links. llms.txt only expresses a priority, to whoever chooses to read it.
What it looks like
The format is markdown, and deliberately simple:
# Lalit on Digital
> AI SEO and GEO consulting. Helping brands become discoverable,
> trusted and cited across AI-powered search.
## Learn
- [What is AI SEO?](https://lalitondigital.com/learn/ai-search/what-is-ai-seo/): How AI search differs from traditional SEO.
- [How AI assistants choose citations](https://lalitondigital.com/learn/ai-search/how-ai-chooses-citations/): What makes a source get quoted.
## Services
- [AI visibility audit](https://lalitondigital.com/services/ai-visibility-audit/): A full review of how AI systems see your site.
## Optional
- [About](https://lalitondigital.com/about/): Background and credentials.Four rules: one H1 with the site name, one blockquote summary, H2 sections containing link lists, and a description after each link. Anything under an ## Optional heading is understood as safe to skip when space is short.
llms-full.txt is the companion: the same content, but with the full text of the pages inlined rather than linked. Useful when you want a model to have everything in one fetch without crawling.
How to write one
- List the pages you would want quoted if someone asked an AI about your subject. Usually 10 to 30.
- Write one sentence per page saying what it answers. Write these for a reader, not for a crawler.
- Group them under H2 headings that match how a person would ask.
- Put anything non-essential under
## Optional. - Save as
llms.txtat your site root, so it resolves atyoursite.com/llms.txt. - Serve it as
text/plainortext/markdown, and make sure it returns a 200.
Twenty minutes, once. Re-check it whenever your important pages change.
Common mistakes
- Listing everything. A file with 400 links expresses no priority and defeats the point.
- Writing descriptions for robots. "AI SEO services company best consultant" helps nobody. Write the sentence you would say to a person.
- Using it for access control. It grants and blocks nothing. That is robots.txt.
- Assuming it replaces a sitemap. Search engines do not read llms.txt.
- Publishing it and never updating it. A stale file pointing at moved pages is worse than none.
- Claiming results from it. If your citations rise after adding llms.txt, something else almost certainly caused it. Do not let anyone sell you the correlation.
When this does not apply
Small sites gain the least. If you have fewer than about ten pages, llms.txt adds nothing that your navigation does not already say, because a model that reaches your home page has effectively already seen your priority list. If your content is behind a login, it cannot help. And if you are hoping for a traffic increase, this is the wrong lever entirely, and the pages under Writing content AI will cite are where the actual work is.
Should you bother?
Yes, but for the right reason and with the right expectation.
Do it because it costs twenty minutes, it cannot hurt you, it forces a genuinely useful exercise in deciding which of your pages matter, and it is immediately useful to any AI agent that is pointed at your site directly. Adoption may well arrive, and the file will already be there.
Do not do it because you expect more citations, more traffic, or a ranking benefit. There is no public evidence for any of those, and anyone presenting llms.txt as an AI-visibility tactic is either mistaken or selling something.
Twenty minutes, no downside, no promises. That is the whole answer.
FAQ
Does Google use llms.txt? Google has not announced support for it, and has publicly compared the idea to the old keywords meta tag. Assume it does not.
Does ChatGPT or Claude use it? Neither OpenAI nor Anthropic has confirmed that their crawlers or browsing tools read llms.txt as a ranking or selection signal. An assistant that is explicitly given the URL will read it, the same way it would read any page you hand it.
Is llms.txt a ranking factor? No. There is no evidence it affects rankings in search or selection in AI answers.
What is the difference between llms.txt and robots.txt? robots.txt controls access, and every major crawler obeys it. llms.txt suggests priority, and no crawler is committed to reading it. They solve unrelated problems.
What is the difference between llms.txt and sitemap.xml? A sitemap lists everything, for search engines, in XML. llms.txt lists what matters most, for language models, in markdown. Search engines read sitemaps. Nobody has confirmed reading llms.txt.
What is llms-full.txt? The same idea with the full page content inlined instead of linked, so a model gets everything in one request.
Which sites publish one? Many developer-facing companies do, particularly documentation sites. Widespread publishing is not evidence of consumption; it mostly reflects how cheap the file is to add.
Can it hurt my site? No, provided the links in it are correct and it is kept current. A file pointing at dead or moved pages is the only real risk.
How do I know whether it is working? You largely cannot, today. Server logs will show whether anything fetched it, and that is the only direct evidence available. Treat any claimed causal effect on citations with suspicion.
Sources
- The original llms.txt proposal, Answer.AI
- Google Search Central commentary on llms.txt
- Your own server logs, which are the only first-party evidence that exists
- First published.