AI
llms.txt
What llms.txt is, why publishers are adopting it, and how it differs from robots.txt and the sitemap.
llms.txt is an emerging convention that gives LLM crawlers a structured, human-readable map of a site's most important content. Proposed by Jeremy Howard (Answer.AI) in late 2024, it sits at the root of a website alongside robots.txt and sitemap.xml. The format is Markdown with an H1 site name, a blockquote description, and H2 sections of bulleted links.
Adoption accelerated through 2025 as AI search engines (Perplexity, ChatGPT Search, Claude's Web Search, Google AI Overviews) signalled they were reading the file. Notifire was one of the early publishers to ship one; this hub tracks the spec's evolution and the publishers and tools that adopt it.
Latest briefings on llms.txt
No briefings on this topic yet. Subscribe to the feed and we’ll cover it as the news cycle moves.
Frequently asked questions
What is llms.txt?
A Markdown file at the root of a website (https://example.com/llms.txt) that lists the site's most important URLs and a short description of each, for LLM crawlers to consume. Format spec at llmstxt.org.
How is it different from sitemap.xml?
Sitemap.xml is an exhaustive machine-readable list of every URL on the site, designed for search-engine crawlers. llms.txt is a curated short list with editorial context (per-link descriptions), designed for LLM ingestion and citation.
Should publishers ship one?
Yes if you want AI assistants to cite you accurately. Without llms.txt, LLMs guess at your structure from crawled HTML; with it, you tell them which pages are authoritative and how to think about each section.
Does Google honour llms.txt?
As of 2026, Google has not formally committed to llms.txt for AI Overviews, but multiple AI search engines (Perplexity, ChatGPT Search, Claude Web Search) confirm they read it. The cost to publish is small enough that publishers don't wait for Google's endorsement.