In short
llms.txt is a plain markdown file at the root of a website that lists its most important pages, so an AI model can find them without crawling everything. It was proposed in September 2024. No major AI company has confirmed using it for ranking or citation, so treat it as cheap documentation, not an SEO tactic.
Key points
- It is a markdown file at yoursite.com/llms.txt: your key pages, one line each, grouped under headings.
- Proposed by Jeremy Howard of Answer.AI in September 2024. Not by OpenAI, Google, Anthropic or Perplexity.
- None of those four has said its crawlers read it as a ranking or citation signal.
- It takes about twenty minutes to write and cannot hurt you if you keep it current.
- Do it for completeness and for agents pointed at your site directly. Do not expect traffic from it.
The short version
llms.txt is a plain markdown file at the root of your site that lists your most important pages, with a sentence about each, so an AI model can find them without crawling everything. It was proposed in September 2024. No major AI company has confirmed using it for ranking or citation, so treat it as cheap, useful documentation rather than an SEO tactic.
Most of what has been written about llms.txt oversells it, usually by people with a tool to sell. This page is short because the honest answer is short.
Where it came from
Jeremy Howard of Answer.AI published the proposal at llmstxt.org on 3 September 2024. It is a community proposal. OpenAI, Anthropic, Google and Perplexity did not write it, did not ask for it, and have not committed to reading it.
That distinction matters more than it sounds. robots.txt works because crawler operators agreed, decades ago, to obey it. sitemap.xml works because the search engines agreed to read it. llms.txt has no agreement behind it yet. It is a sensible idea waiting for someone to adopt it.
Google's John Mueller said as much in April 2025: no AI system he knew of was using llms.txt, and he compared it to the keywords meta tag, which sites filled in for years after search engines had stopped reading it.
The three files, and what each actually does
| File | Answers the question | Honoured by |
|---|---|---|
robots.txt | "Are you allowed in?" | Every major crawler |
sitemap.xml | "What pages exist?" | Search engines |
llms.txt | "Which pages actually matter?" | No confirmed consumer yet |
The common mistake is treating llms.txt as a replacement for one of the others. It is not. If you want to control AI crawler access, that is robots.txt. If you want your pages discovered, that is sitemap.xml and internal links. llms.txt only expresses a preference, to whoever chooses to read it.
What it looks like
The format is markdown, and deliberately plain. This is a trimmed version of the file on this site, which you can read at lalitondigital.com/llms.txt:
# Lalit Patel
> SEO and performance marketing specialist, Mumbai, 12+ years in search.
> Now focused on AI search: GEO, AEO and AI SEO.
## Learn
- [Learn AI](https://lalitondigital.com/learn/): A free course in using AI well, and in being found by it.
- [Getting found by AI](https://lalitondigital.com/learn/getting-found/): Access, readability and discovery for AI crawlers.
## Profile
- [Lalit Patel](https://lalitondigital.com/): Full professional profile, expertise and FAQ.
## Optional
- [Extended profile](https://lalitondigital.com/llms-full.txt): The complete profile text in one file.Four rules. One H1 with the site name. One blockquote summary. H2 sections containing link lists. A description after each link. Anything under an ## Optional heading is understood to be safe to skip when space is short.
llms-full.txt is the companion file: the same pages, but with their full text inlined rather than linked. Useful when you want a model to have everything in one fetch.
How to write one
- List the pages you would want quoted if someone asked an AI about your subject. Usually ten to thirty. If you cannot choose, that is the real problem, and no file fixes it.
- Write one sentence per page saying what question it answers. Write it for a person. You are describing the page, not stuffing it.
- Group the links under H2 headings that match how someone would ask.
- Put anything non-essential under
## Optional. - Save it as
llms.txtat the site root, so it resolves atyoursite.com/llms.txt. - Serve it as
text/plainortext/markdown, and check it returns a 200 rather than your 404 page with a 200 status, which is more common than it should be.
Twenty minutes, once. Re-read it whenever your important pages change.
Common mistakes
- Listing everything. A file with 400 links expresses no priority at all. That is a sitemap with worse formatting.
- Writing descriptions for robots. "AI SEO services company best consultant Mumbai" helps nobody. Write the sentence you would say to a person who asked what the page was about.
- Using it for access control. It grants nothing and blocks nothing. That job belongs to robots.txt and your firewall.
- Assuming it replaces a sitemap. Search engines do not read llms.txt. Keep the sitemap.
- Publishing it and never looking at it again. A stale file pointing at moved pages is worse than no file.
- Claiming results from it. If your citations rose after you added llms.txt, something else caused it. Anyone selling you the correlation is selling you something.
When this does not apply
Small sites gain the least. If you have fewer than about ten pages, llms.txt adds nothing your navigation does not already say. A model that reaches your home page has already seen your priority list. If your content sits behind a login, the file cannot help. And if what you want is more traffic, this is the wrong lever entirely. The pages under Writing content AI will cite are where that work happens.
Should you bother?
Yes, but for the right reason and with the right expectation.
Do it because it costs twenty minutes, it cannot hurt you, and it forces a useful decision about which of your pages matter. It is also immediately useful to any AI agent that is handed your URL directly, which happens more now that people paste sites into assistants. If adoption arrives, the file is already there.
Do not do it because you expect more citations, more traffic or a ranking benefit. There is no public evidence for any of those. Anyone presenting llms.txt as an AI visibility tactic is either mistaken or selling something.
Twenty minutes, no downside, no promises. That is the whole answer.
FAQ
Does Google use llms.txt? No. Google has not announced support for it. In April 2025 John Mueller of Google said no AI system he knew of was using it, and compared it to the keywords meta tag. Assume Google ignores it.
Does ChatGPT or Claude use it? Neither OpenAI nor Anthropic has said that its crawlers or browsing tools read llms.txt as a ranking or selection signal. An assistant that is given the URL directly will read it, the same way it would read any page you hand it.
Is llms.txt a ranking factor? No. There is no evidence that it affects rankings in search or which sources are chosen in AI answers.
What is the difference between llms.txt and robots.txt? robots.txt controls access and every major crawler obeys it. llms.txt suggests priority and no crawler has committed to reading it. They solve unrelated problems and you need the first one far more than the second.
What is the difference between llms.txt and sitemap.xml? A sitemap lists every page, for search engines, in XML. llms.txt lists the pages that matter most, for language models, in markdown with a sentence per page. Search engines read sitemaps. Nobody has confirmed reading llms.txt.
What is llms-full.txt? The same idea with the full text of each page inlined instead of linked, so a model gets everything in one request instead of following links.
Which sites publish one? Mostly developer documentation. Anthropic's documentation, Cloudflare's developer docs and every site hosted on Mintlify have one, and Mintlify generates it automatically. That is a sign of how cheap the file is to add, not evidence that anyone consumes it.
Can it hurt my site? No, as long as the links in it are correct and you keep it current. A file pointing at dead or moved pages is the only real risk.
How do I know whether it is working? You largely cannot, today. Your server logs will show whether anything fetched the file, and that is the only direct evidence there is. Treat any claimed effect on citations with suspicion until one of the AI companies says it reads the file.
Sources
- The llms.txt proposal, Jeremy Howard, Answer.AI, 3 September 2024
- Anthropic's llms.txt and Cloudflare's llms.txt, two live examples
- This site's llms.txt, the file the example above is trimmed from
- Your own server logs, which are the only first-party evidence that exists
- Rewritten. Added dated sources, named examples of sites that publish one, and this site's own file as a worked example.
- First published.