Learn how to implement llms.txt in your website or documentation
What is llms.txt?
The llms.txt file is a proposed standard that helps AI models better understand and interact with your website's content. Unlike robots.txt or sitemap.xml, it's specifically designed to enhance AI interactions by providing structured content overviews and navigation paths.
Purpose and Benefits
- Enhanced AI Understanding: Provides a structured overview of your website content, helping AI systems understand your site quickly and accurately
- Efficient Information Retrieval: Enables AI systems to locate and retrieve relevant information efficiently
- Improved Contextual Understanding: Reduces misinterpretation by providing context and relationships between content
- Better User Experience: Leads to more accurate, context-aware responses when users interact with AI about your content
- Overcoming Context Limitations: Helps AI systems work within their context window limitations by providing streamlined content access
How llms.txt Differs from robots.txt and sitemap.xml
While these files might seem similar, they serve distinct purposes:
- robots.txt: Focuses on controlling search engine crawler access but doesn't help with content understanding
- sitemap.xml: Lists all indexable pages without providing context or helping with content processing
- llms.txt: Specifically addresses AI-related challenges by providing content structure in a format optimized for AI processing
The llms.txt Specification
The specification defines two distinct files:
- /llms.txt: A streamlined view of documentation navigation to help AI systems quickly understand a site's structure
- /llms-full.txt: A comprehensive file containing all documentation in one place
Both files use Markdown formatting, which provides a natural hierarchy that AI models can easily parse.
Basic Implementation Guide
1. Create the File Structure
The llms.txt file uses Markdown with a specific structure:
# Your Website/Project Name
> A brief description of your website or project
## Documentation
- [Getting Started](/docs/getting-started) - Guide for new users
- [API Reference](/docs/api) - Complete API documentation
- [Tutorials](/docs/tutorials) - Step-by-step guides
## Examples
- [Basic Implementation](/examples/basic) - Simple integration example
- [Advanced Features](/examples/advanced) - Using advanced capabilities
## Optional Resources
- [Community Forum](/community) - Get help from other users
- [Change Log](/changelog) - Track updates and changes
2. Place the File in the Correct Location
Save the file as llms.txt in your website's root directory, ensuring it's accessible at yourwebsite.com/llms.txt. If you're creating a comprehensive version with all your documentation, save it as llms-full.txt.
3. Add HTTP Headers (Optional but Recommended)
Add the following HTTP header to your server configuration:
X-Robots-Tag: llms-txt
4. Verify Implementation
Test your implementation by:
- Accessing
yourwebsite.com/llms.txt - Checking HTTP headers
- Validating the file format
Using llms.txt with AI Systems
Currently, most AI models don't automatically discover and index llms.txt files. To use your llms.txt file with AI systems:
- Direct Link: Provide the AI with a link to your llms.txt file
- Manual Copy: Copy the contents of your llms.txt file directly into your prompt
- File Upload: Use the AI tool's file upload feature if available
As adoption increases, more AI systems will likely integrate automatic discovery of llms.txt files.
Where To Find a list of all llms.txt files?
llms.txt hub is rapidly becoming a popular open-source hub tracking all the websites that have implemented the llms.txt and llms-full.txt standards. Updated regularly, this comprehensive directory allows you to:
- Discover real-world implementations across various industries and platforms
- Study how leading organizations structure their llms.txt files
- Access the latest news and developments related to the standard
- Find open-source tools, plugins, and resources for implementing llms.txt on your own website
By exploring this directory, developers and content creators can observe implementation patterns, stay informed about best practices, and connect with the growing community of websites optimizing their content for AI systems through the llms.txt standard.
Generation Tools
Several tools can help you generate llms.txt files:
-
llmstxt by dotenv: Open source CLI tool that generates
llms.txtbased on a site'ssitemap.xmlfile. - llmstxt by Firecrawl: Uses Firecrawl to generate a llms.txt file.
- Mintlify: Documentation platform with llms.txt generation
Real-World Examples
Many organizations have already adopted the llms.txt proposed standard:
- Cloudflare: Impressive and extensive documentation about all the Cloudflare services
- Anthropic's llms.txt: All their documentation and prompt library
- Perplexity's llms-full.txt: A comprehensive implementation all their documentation and content website
- ElevenLabs: All the documenation about their API and product guides
Best Practices
Keep it Updated: Regularly update your llms.txt file as your website structure changes to ensure AI systems have the most current information.
-
Use Clear Markdown Structure:
- Start with an H1 project name
- Include a blockquote summary
- Use H2 headers to organize documentation links
- Provide brief descriptions for each link
Be Selective: Focus on the most important resources in llms.txt, using the Optional section for less critical content.
Test with AI Systems: Verify that AI models correctly interpret and navigate your content based on your llms.txt file.
Optimize for AI Processing: Remove non-essential markup and scripts in llms-full.txt to help AI models focus on the important content.
Top comments (1)
The llms.txt file is a proposed standard designed to help AI systems better understand and navigate your website’s content by providing a structured overview. Unlike robots.txt or sitemap.xml, which focus on indexing and crawling, llms.txt enhances AI interaction by offering organized content with clear navigation paths. It improves AI’s ability to retrieve relevant information, understand context, and deliver more accurate responses. Implementing this file involves using a Markdown structure to define key sections, and it should be kept updated to reflect any changes to your website. This helps AI systems process and interpret your content efficiently.