What Is LLMs.txt and How to Get Started

As AI continues to grow, boosting your content’s AI readability has become more important. Large language models (LLMs) are powerful tools, but they can struggle with processing complex websites.

This is where llms.txt comes in. It’s a simple file designed to help AI systems better understand your web content.

In this guide, we will learn what an llms.txt file is, explore its uses, and best practices to use llms.txt effectively.

What is llms.txt?

The llms.txt file is a proposed standard designed to improve how AI models interact with and understand your website’s content. It acts as a curated index for large language models (LLMs), offering brief summaries of the site's purpose, important contextual information, and prioritized links.

AI systems face limitations with context windows, making it hard to process large documentation sites effectively.

Traditional SEO methods are designed for search crawlers, not reasoning engines, so they don’t resolve this challenge. When AI systems attempt to process HTML pages directly, they get overwhelmed by navigation elements, JavaScript, CSS, and other unnecessary information, leaving less room for the actual content.

Jeremy Howard, co-founder of Answer.AI, introduced llms.txt to address this technical issue. llms.txt tackles this problem by providing the AI with the precise information it needs in a format it can easily process.

Major platforms such as Mintlify, Cloudflare, and Cursor have integrated llms.txt, highlighting its effectiveness. For a comprehensive list of websites that have adopted the llms.txt standard, you can visit the llms.txt directory, which lists around 474 companies.

But why should your content be LLM-friendly? Let's find out.

Why is llms.txt beneficial?

llms.txt provides a host of benefits. Let's take a look at them.

• Efficient information retrieval: It helps AI locate relevant information quickly.

• Enhanced AI understanding: It offers AI systems a clear, structured view of your website's content, enabling faster and more accurate understanding.

• Better user interaction: It ensures that AI provides more relevant and context-aware responses when users inquire about your site.

One day, every company will need two versions of their docs: one for humans and another for LLMs.

Why we made the leap in providing /llms.txt for all customers: https://t.co/p8IhlFJI2E pic.twitter.com/uK9fX44U4x

— Mintlify (@mintlify) November 20, 2024

llms.txt VS robots.txt VS sitemap.xml

llms.txt serves a completely different purpose compared to established web standards such as sitemap.xml and robots.txt. Let’s take a look at the major differences between them.

Feature	llms.txt	robots.txt	sitemap.xml
Purpose	Guides AI models on how to use a site's content.	Tells web crawlers which pages they can and cannot access.	Lists URLs to help search engines index a site efficiently.
Audience	LLMs (Large Language Models) like ChatGPT, Bard, etc.	Search engine crawlers (Googlebot, Bingbot, etc.).	Search engines(Google, Bing, etc.).
Content analysis	AI may process and analyze content based on rules.	Only provides crawl instructions; does not process content.	Only lists URLs; does not process content.

Structure of an llms.txt file

Before we start to understand the structure of an llms.txt file, you should know what markdown files are.

Markdown files (.md files) are lightweight, plain-text documents formatted using Markdown, a simple markup language designed for easy readability and conversion to structured documents like HTML, PDFs, and rich text. Markdown works on any operating system and is supported in apps like GitHub, Notion, Obsidian, VS Code and Jupyter Notebook.

Now, let’s come back to the structure of an llms.txt file. It is a markdown file with a specific structure. The specification defines two distinct files:

1. llms.txt

2. llms-full.txt

Let’s go through them in detail.

1. llms.txt

The file should begin with an H1 header containing the project name, followed by a blockquote that provides a summary. The rest of the document should be organized using H2 headers to categorize the documentation links. The "Optional" section should be used to highlight less essential resources. This is what it would look like.

# Project Name
> Brief project summary
Additional context and important notes
## Core Documentation
- [Quick Start](URL): Description of the resource
- [API Reference](URL): API documentation details
## Optional
- [Additional Resources](URL): Supplementary information


          
      
          
      

2. llms-full.txt

This file includes the entire documentation content in markdown format. Let’s take a look at the snippet given below for a better understanding of llms-full.txt.

# Breaking Changes
Source: https://www.activepieces.com/docs/about/breaking-changes
This list shows all versions that include breaking changes and how to upgrade.

## 0.46.0

### What has changed?

* The UI for "Array of Properties" inputs in the pieces has been updated, particularly affecting the "Dynamic Value" toggle functionality.
### When is action necessary?  

* No action is required for this change.
* Your published flows will continue to work without interruption.
* When editing existing flows that use the "Dynamic Value" toggle on "Array of Properties" inputs (such as the "files" parameter in the "Extract Structured Data" action of the "Utility AI" piece), the end user will need to remap the values again.
* For details on the new UI implementation, refer to this [announcement](https://community.activepieces.com/t/inline-items/8964).

## 0.38.6

### What has changed?
* Workers no longer rely on the `AP_FLOW_WORKER_CONCURRENCY` and `AP_SCHEDULED_WORKER_CONCURRENCY` environment variables. These values are now retrieved from the app server.

### When is action necessary? 

* If `AP_CONTAINER_TYPE` is set to `WORKER` on the worker machine, and `AP_SCHEDULED_WORKER_CONCURRENCY` or `AP_FLOW_WORKER_CONCURRENCY` are set to zero on the app server, workers will stop processing the queues. To fix this, check the [Separate Worker from App](https://www.activepieces.com/docs/install/configuration/separate-workers) documentation and set the `AP_CONTAINER_TYPE` to fetch the necessary values from the app server. If no container type is set on the worker machine, this is not a breaking change.  


          
      
          
      

How to add llms.txt on your website

To add an llms.txt file to your website, here's a detailed breakdown:

1. Create the file structure: This file should contain a list of links to key pages and resources on your website that you want LLMs to focus on. Use Markdown format for easy readability and parsing by tools.
If you want to include a comprehensive version of your documentation, create a file named llms-full.txt. Having both of the files can give better results.

2. Upload to your website's root directory: Ensure the file is accessible at yourwebsite.com/llms.txt.
If using a CMS, upload the file to the root directory or a designated location accessible via a URL. If required, you can also put it in the public directory.

3. Add HTTP headers (Optional): Add the following HTTP header to your server configuration: X-Robots-Tag: llms-txt. This can help bots understand that the file is intended for LLMs. You can modify robots.txt to allow/disallow the LLM agents.

Here's an example that allows access for AI search/agents but disallows training data collection:

# Allow AI search and agent use

User-agent: OAI-SearchBot

User-agent: ChatGPT-User

User-agent: PerplexityBot

User-agent: FirecrawlAgent

User-agent: AndiBot

User-agent: ExaBot

User-agent: PhindBot

User-agent: YouBot

Allow: /

# Disallow AI training data collection

User-agent: GPTBot

User-agent: CCBot

User-agent: Google-Extended

Disallow: /

          
      
          
      

4. Verify implementation: Open yourwebsite.com/llms.txt in your browser to ensure it's accessible. Verify that the robots.txt header is set correctly (if you've added it). Use a tool or validator to ensure the llms.txt file is in the correct Markdown format.

Tools for generating llms.txt files

Several tools can help generate llms.txt files. Below, we have listed a few of them:

• llmstxt by dotenv: Creates llms.txt based on your sitemap.xml. Learn more at llmstxt by dotenv.
• llmstxt by Firecrawl: Uses Firecrawl to generate an llms.txt file. More details at llmstxt by Firecrawl.
• Mintlify: Automatically generates both llms.txt and llms-full.txt for hosted documentation. Read about it at Mintlify.

How to use llms.txt with AI systems

Unlike search engines that actively crawl the web, current LLMs don’t automatically discover and index llms.txt files. You must manually provide the file content to your AI system. This can be done by pasting the link, copying the file contents directly into your prompt, or using the AI tool’s file upload feature.

Here's how different platforms use llms.txt.

ChatGPT

You can paste the URL or full content of the llms-full.txt file into ChatGPT, allowing the model to access relevant details. Then, you can ask specific questions about what you’d like to know.

Claude

Since Claude does not have built-in browsing capabilities, users can manually paste the content or upload the file to provide the necessary context. This method ensures Claude's responses are based on accurate and up-to-date information, improving reliability.

Cursor

Cursor allows users to add and index third-party documentation, making it available as context in conversations. To do this, type @Docs > Add new doc, which will open a modal. Here, you can insert a link to the llms-full.txt file, allowing it to be used as context like any other document.

Best practices for using llms.txt effectively

By implementing these best practices, you can enhance the AI's ability to understand and interact with your content in the most effective way possible.

1. Organize content for AI understanding: Ensure that the file contains content in a format that’s easy for AI systems to process. Organize the content logically with clear headings, subheadings, and bullet points.

2. Prioritize important information: Only include the most relevant and necessary content in llms.txt. Avoid cluttering the file with redundant data, as this can take up valuable context space for AI systems.

3. Update regularly: Keep llms.txt up to date to reflect any changes in the content or structure of your website. This ensures the AI always has access to the most current information, reducing the risk of outdated processing.

4. Provide contextual metadata: If necessary, include metadata or annotations that provide context for the content. This could be descriptions, summaries, or links that help the AI system understand the broader context of the information.
For a better understanding, refer to the example given below.

# MyWebsite.com

> Automation tools for email marketing.
## Documentation

- **[Getting Started](/docs/getting-started)** – Quick setup instructions.
## AI Access Policy

- **Allowed**: `/docs/` *(Public content accessible to AI)*
- **Restricted**: `/private/` *(Do not index or use for AI training)*
- **Training**: `disallow`

## Metadata

- **Last Updated**: 2025-03-27
- **Primary Language**: English
- **Related**: [Mailmodo Help Center](https://support.mailmodo.com/), [Email Automation Guide](/blog/email-automation)


          
      
          
      

5. Test and validate: Regularly test how AI systems interact with llms.txt to ensure the file serves its intended purpose. Check for any issues or inefficiencies in the way the content is being processed and make adjustments accordingly.

Way forward

What’s next for llms.txt? It marks a move toward AI-first documentation.

Unlike traditional SEO techniques focusing on search crawlers, llms.txt is specifically structured to assist AI in bypassing unnecessary elements like HTML, JavaScript, and CSS. Instead, it presents the relevant content in a simplified, structured format, enabling AI to generate insightful responses.

As more websites adopt this file, we can expect the development of new tools and best practices to make content accessible to humans and AI assistants.

Resources

Explore these key resources to understand, implement, and stay updated on the llms.txt standard.

• llms.txt Standard Proposal: Official proposal outlining the llms.txt standard and its implementation.
• llms.txt Reddit Community: A discussion hub for updates, insights, and community discussions on llms.txt.
• llms.txt Hub GitHub Repository: An open-source repository with tools and resources for llms.txt adoption.
• llms.txt Directory: A curated list of websites that have implemented llms.txt.