# llms.txt: The Complete Guide for AI Crawlers in 2026
llms.txt is a proposed IETF standard (draft-nottingham-llm-01) — a simple Markdown text file placed at your domain root that tells AI crawlers which pages matter, how they relate, and what each page contains. Think of it as robots.txt for AI: instead of blocking crawlers, it gives them a structured, low-noise map of your content. Setting it up takes 10 minutes and can dramatically improve how ChatGPT, Perplexity, and Claude extract and cite your content.
| Feature | llms.txt | robots.txt | Sitemap.xml |
|---|---|---|---|
| Purpose | Include content for AI | Block crawler paths | List all pages for indexing |
| Audience | AI/LLM crawlers | All crawlers | Search engines |
| Format | Markdown headings + links | Allow/Disallow directives | XML URL list |
| Tokens consumed | 300-500 | Varies | Varies |
| Avg setup time | 10 minutes | 5 minutes | Automated |
Here is exactly what determines whether AI engines cite your content — and how to fix the gaps.
Why llms.txt Matters Now
When AI systems visit your website, they don't see your beautiful design. They don't experience your carefully crafted UX. They see raw HTML — thousands of lines of div soup, navigation elements, and footer content that obscures the actual information they're looking for. The llms.txt standard changes that.
“**Related:** [AI Citations The Complete Guide to Getting Your Website](/blog/ai-citations-complete-guide-2026) — actionable guide with step-by-step instructions.”
“**Related:** [AI SEO Audit Complete 2026 Guide to Find and Fix AI Cit](/blog/ai-seo-audit-tool) — actionable guide with step-by-step instructions.”
“**Related:** [Best SEO Tools for Perplexity in 2026 The Complete Guid](/blog/best-seo-tools-for-perplexity) — actionable guide with step-by-step instructions.”
“**Related:** [EEAT 2026 The Complete Guide to Building Trust Signals ](/blog/eeat-2026-complete-guide) — actionable guide with step-by-step instructions.”
“**Related:** [Entity SEO The Complete 2026 Guide to Knowledge Graph O](/blog/entity-seo) — actionable guide with step-by-step instructions.”
- What Is llms.txt?
- Why llms.txt Matters Now
- How to Create an llms.txt File
What Is llms.txt?
llms.txt is a proposed IETF standard (draft-nottingham-llm-01) — a simple text file placed at the root of your domain that provides AI systems with a structured summary of your site's content. Think of it as robots.txt for AI: instead of telling crawlers what NOT to access, llms.txt tells them what TO access and how to understand it.
The standard was proposed by Mark Nottingham, a well-known IETF contributor behind the HTTP/2 spec and robots.txt itself. The format is deliberately simple: Markdown-like structure with section headings and links. There's no complex schema, no authentication requirements, no API calls. You write it once and update it when your content organization changes.
Why llms.txt Matters Now
AI-powered search and browsing are consuming web content at unprecedented scale. When ChatGPT browses the web, when Perplexity researches a topic, when Claude analyzes documentation — they all encounter the same problem: extracting signal from noise.
A typical web page produces 8,500+ tokens of HTML. After stripping navigation, headers, footers, scripts, and styles, perhaps 400 tokens are actual content. The rest is waste — computational overhead that slows AI systems and reduces the accuracy of content extraction.
A well-structured llms.txt produces 300-500 tokens of pure, structured information. Every token is signal. No waste.
How to Create an llms.txt File
### Step 1: Create the File Place a file named `llms.txt` at your domain root: `https://yoursite.com/llms.txt`
### Step 2: Structure Your Content ```markdown # Your Site Name > Brief description of what your site is and who it's for.
## Key Pages - [Homepage](https://yoursite.com) — Main landing page - [About](https://yoursite.com/about) — Company information and team - [Blog](https://yoursite.com/blog) — Articles and insights - [Pricing](https://yoursite.com/pricing) — Plans and features - [Contact](https://yoursite.com/contact) — Get in touch
## Documentation - [API Reference](https://yoursite.com/docs/api) - [Getting Started Guide](https://yoursite.com/docs/getting-started)
## For AI Systems - Sitemap: https://yoursite.com/sitemap.xml - Content updates: Weekly ```
### Step 3: Add the Companion File Create `llms-full.txt` at the same location with expanded content — more detail, more links, more context. This is what AI systems read when they need deeper understanding.
### Step 4: Verify Test with: `curl https://yoursite.com/llms.txt` — it should return clean Markdown, not HTML.
Best Practices
Keep it concise. The standard recommends keeping llms.txt under 200 lines. Think of it as a table of contents, not a full sitemap.
Update it when your site changes. An outdated llms.txt is worse than no llms.txt because it misdirects AI systems.
Use descriptive link text. Instead of "Click here," write "API Reference Documentation." AI systems use link text as context for understanding what each page contains.
Add an llms-full.txt for depth. Sites with 50+ pages should include the expanded version with categorized content, deprecation notes, and priority signals.
Pair with sitemap_agentic_discovery.xml. Shopify stores automatically generate this — it tells AI agents about your llms.txt and agents.md files. If you're on another platform, create it manually.
Common Mistakes
Creating llms.txt without maintaining it. An llms.txt that references pages that no longer exist is actively harmful — AI systems will attempt to access dead links.
Making it too long. If your llms.txt is 1,000 lines, you've defeated the purpose. AI systems need a quick summary, not a novel.
Forgetting to list deprecations. If you moved or removed content, note it explicitly. AI systems cache content and may continue referencing old URLs without explicit deprecation signals.
Verification Checklist
- [ ] File accessible at `https://yoursite.com/llms.txt`
- [ ] Returns `Content-Type: text/plain` or `text/markdown`
- [ ] Contains site title and description
- [ ] Lists 5-20 key pages with descriptive anchor text
- [ ] Links to sitemap.xml
- [ ] Updated within the last 90 days
- [ ] No broken internal links
Related Articles
- [How AI Citation Algorithms Work — The Technical Deep Dive](/blog/how-ai-citation-algorithms-work)
- [Technical SEO for AI Search: The Complete Implementation Guide (see also our [llms.txt guide](/blog/llms-txt-what-it-is-why-you-need-it))](/blog/technical-seo-ai-search-guide)
- [Technical SEO for AI Search: The 2026 Complete Guide](/blog/technical-seo-ai-search-2026)
FAQ
Q: What is llms.txt and do I need it?
A: llms.txt is a proposed IETF standard — a simple text file at your domain root that tells AI systems what your site contains and how to navigate it. Think of it as robots.txt for AI. Every website should have one. It takes 10 minutes to create and significantly improves how AI crawlers understand your content.
Q: Which AI crawlers should I allow in robots.txt?
A: At minimum, allow: GPTBot (ChatGPT), ChatGPT-User (real-time search), Google-Extended (Gemini), CCBot (Common Crawl), PerplexityBot, ClaudeBot (Anthropic), and Applebot-Extended. Blocking any of these reduces your visibility on the corresponding AI platform.
Q: How do I check if my structured data is correct?
A: Use Google's Rich Results Test (search.google.com/test/rich-results) or Schema.org's validator. At minimum, every page should have: Organization schema on the homepage, Article/BlogPosting schema on content pages, and FAQPage schema on pages with FAQ content. Read our complete JSON-LD recommendations guide for detailed requirements.
Q: Do I need to optimize for Core Web Vitals for AI search?
A: Yes — but with different priorities. AI crawlers have shorter timeout windows than Googlebot, so TTFB (Time to First Byte) under 1 second is critical. Content must appear in raw HTML without JavaScript execution. Mobile-first indexing is increasingly used by AI crawlers. Traditional CWV metrics (LCP, INP, CLS) are secondary to content accessibility and rendering speed.
Q: What technical issues most commonly block AI visibility?
A: The top technical blockers are: (1) AI crawlers blocked in robots.txt, (2) Content rendered exclusively via JavaScript (invisible to most AI crawlers), (3) Missing structured data, (4) Slow server response times (TTFB > 2 seconds), and (5) No llms.txt file. Run a free audit at geoxylia.com to check all of these automatically.
Run a free AI Citability Audit at geoxylia.com/audit to verify your llms.txt is properly configured and discoverable by all major AI platforms. The audit checks llms.txt completeness, AI crawler access, and 8 other dimensions of AI visibility.
Further Reading
Continue exploring this topic with these related deep dives:
- [llms.txt and Content Strategy: Making AI-Ready Content for 2026](/blog/llms-txt-and-content-strategy-making-ai-ready-content)
Further Reading
Continue exploring this topic with these related deep dives:
- [llms.txt: The Complete Technical Setup Guide for 2026](/blog/llms-txt-the-complete-technical-setup-guide)