Files at the root of your site

What llms.txt is, and which crawler switch does what

llms.txt is a proposal for a short markdown file that points an agent at the pages worth reading. It is not a standard from a standards body, and it is not how you opt in or out of ChatGPT, Perplexity, or Google AI Overviews. Those products document robots.txt tokens. The tokens do different jobs.

What the proposal is

The page at llmstxt.org calls /llms.txt a proposal to standardise on a file that helps agents use a website. The page lists 3 September 2024 as the original publication and 10 August 2026 as an update. The current text is marked v2.

The file is markdown. The only required piece is an H1 with the name of the site or project. A blockquote can summarize it. Later H2 sections can list links, each one a markdown link and an optional note. The proposal says the file should stay small enough to read in one context window, and the detail should live behind the links. It also suggests offering a clean markdown version of important pages, and pointing at the file with a rel="describedby" link.

It is meant to sit beside robots.txt and sitemap.xml, not replace them. In the proposal’s own comparison, a sitemap lists pages for search engines, llms.txt is a curated overview, and robots.txt says what automated access is acceptable. The proposal says the file was expected to be used when an agent needs information while helping someone, rather than as a training dump. It says that is how it has been used, and that a training run could still read it.

A minimal file looks like this. It is an illustration of the shape, not a file you must publish:

# Your product

> One sentence: what it is, who it is for, what it costs.

## Pages

- [Pricing](https://example.com/pricing): What a buyer pays.
- [FAQ](https://example.com/faq): The questions a buyer actually asks.

How much adoption to believe

The proposal page says thousands of sites publish a file, that some documentation tools generate one, and that OpenAI, Anthropic, and Gemini publish llms.txt files for their own developer docs. It also says Chrome’s Lighthouse checks for one. That is the proposal’s account. It is not a statement from Google Search, from OpenAI’s crawler docs, or from Perplexity that they read /llms.txt when deciding which product to recommend to a buyer.

Google’s optimization guide says you do not need new machine-readable files, AI text files, markup, or markdown to appear in Google Search, including its generative AI features, because Google Search does not use them. It says maintaining an llms.txt file for other systems is fine, and that doing so will neither harm nor help your site’s visibility or rankings in Google Search, because Search ignores those files. The same guide tells site owners they can ignore “creating unnecessary AI text files (like llms.txt)” as a Google Search tactic.

OpenAI’s crawler page explains robots.txt tokens. It does not list llms.txt as a requirement for ChatGPT search. Perplexity’s crawler page does not either. The older OpenAI address platform.openai.com/docs/bots redirects to the current crawler page.

A file can still be worth having if you want a clean summary for an agent that chooses to fetch it, especially for documentation. Citebase publishes one at /llms.txt. It describes the product and links the sample. Publishing it does not cause an assistant to recommend Citebase. Nothing on llmstxt.org says a file would do that.

If you write one, match the pages a buyer would trust: pricing, what it is, who it is for. Keep it the same as the site. An llms.txt that contradicts the homepage is a second, stale pitch.

robots.txt is the control these companies document

robots.txt is a text file at the root of your site. A group starts with User-agent and then Allow or Disallow. Put a group of its own for each token you care about, so you are not guessing how a wildcard group applies. Then open https://yourdomain/robots.txt in a browser and read what you actually shipped.

A firewall or bot fight in front of the site can block a crawler even when robots.txt allows it. Perplexity documents that case for people using Cloudflare or AWS WAF. Check the host before you assume the text file is the reason you are absent.

GPTBot, OpenAI, training

OpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot indicates the content should not be used for that training. It is not the token for ChatGPT search.

OAI-SearchBot, OpenAI, search

OpenAI says OAI-SearchBot surfaces websites in ChatGPT’s search features. If you opt out, the site will not be shown in ChatGPT search answers. It can still appear as a navigational link. They recommend allowing it if you want to be in those results, and they say a robots.txt change can take about 24 hours to show up for search.

The two settings are independent. OpenAI’s example is: allow OAI-SearchBot so you can appear in search, and disallow GPTBot so the content is not used for training. If you allow both, they may use one crawl for both purposes so they do not crawl twice.

There is also OAI-AdsBot, for landing pages submitted as ChatGPT ads. It is not the search opt-out. And there is ChatGPT-User, for some visits a person triggers inside ChatGPT. OpenAI says ChatGPT-User is not automatic crawling, that robots.txt rules may not apply, and that it is not what decides search inclusion. Search opt-out is OAI-SearchBot.

PerplexityBot and Perplexity-User

Perplexity says PerplexityBot surfaces and links sites in Perplexity search results and is not used to crawl content for AI foundation models. They recommend allowing it, and allowing their published IP ranges, if you want the site in search results. They say each robots.txt setting is independent and can take up to 24 hours.

Perplexity-User is the fetch that happens because a person asked a question. Perplexity says it is not used for web crawling or for training foundation models, and that because a user requested the fetch, this fetcher generally ignores robots.txt. Their docs do not say that a Disallow of PerplexityBot is a total blackout of every Perplexity request. They do say PerplexityBot is the agent for search results, and they tell you to allow it if you want to be surfaced there.

Google-Extended is not the AI Overview switch

Google’s crawler reference says Google-Extended has no separate HTTP user agent. Existing Google crawlers fetch the pages. The token is only a control: whether content Google has crawled may be used to train future Gemini models that power Gemini Apps and Vertex AI, and for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal there.

AI Overviews are a Search feature. The crawl control Google names for Search is Googlebot. The on-or-off control for appearing in AI Overviews, AI Mode, and generative AI features in Discover is the Search generative AI setting in Search Console, described in the AI Overviews guide. Blocking Google-Extended is a training and grounding choice. It is not how you enter or leave an AI Overview.

The tokens on one table

A block you can read, not a block you must paste

This illustration allows the search crawlers and leaves training as its own choice. It is not Citebase’s file, and it is not a requirement from any of these companies. Read your whole robots.txt before adding a group. If Googlebot is already disallowed, the site is out of ordinary Google Search, not only out of AI answers. Swap the GPTBot and Google-Extended lines to Allow if you want those crawls as well. The point of the block is that the lines are separate choices.

# ChatGPT search. Separate from training.
User-agent: OAI-SearchBot
Allow: /

# Perplexity search results.
User-agent: PerplexityBot
Allow: /

# Google Search, including AI Overviews.
User-agent: Googlebot
Allow: /

# OpenAI training. Independent of OAI-SearchBot.
User-agent: GPTBot
Disallow: /

# Gemini training and grounding. Not Google Search inclusion.
User-agent: Google-Extended
Disallow: /

Whether you disallow GPTBot or Google-Extended is a policy choice about training and grounding. OpenAI wrote the two OpenAI tokens so you can allow search and refuse training. This guide does not tell you which training choice to make. It tells you not to treat them as the same switch.

This site’s robots.txt allows GPTBot, PerplexityBot, and Google-Extended by name, and its wildcard group allows other agents except /thanks. OAI-SearchBot is not named, so the wildcard group is what applies to it. That is a crawl preference for getcitebase.com. It is not a citation result.

What to do this week

1. Open your robots.txt. Find GPTBot, OAI-SearchBot, PerplexityBot, Googlebot, and Google-Extended. If a token has no group of its own, the User-agent: * group is the one to read.
2. In Search Console, open Settings and then Search generative AI, and see whether the property is included, excluded, or inheriting. The AI Overviews guide has the official meaning of each.
3. If you want a summary file for agents, add an llms.txt that matches your pricing page. Do not expect it to change an AI Overview. Google says Search ignores it.
4. Run the manual check. The file is not a substitute for reading the answer.

Sources

• llmstxt.org, the llms.txt proposal (v2; the page lists 3 September 2024 and 10 August 2026).
• Google Search Central, Optimizing your website for generative AI features on Google Search.
• Google Search Central, robots.txt introduction and AI features and your website.
• Google, Google-Extended.
• Google Search Console Help, Search generative AI control.
• OpenAI, Overview of OpenAI Crawlers (platform.openai.com/docs/bots redirects there).
• Perplexity, Perplexity Crawlers.