Key Takeaways: An
llms.txtfile is a markdown file hosted at your domain root (/llms.txt) designed to give AI web crawlers a curated, noise-free index of your key pages. Here is our exactllms.txtfile annotated line-by-line.
As artificial intelligence models increasingly crawl the web for retrieval-augmented generation (RAG), traditional HTML web pages pose challenges. Heavy JavaScript bundles, navigation menus, and inline tracking scripts pollute context windows and increase processing latency.
The proposed llms.txt standard solves this by providing a clean, human-readable markdown file at https://yourdomain.com/llms.txt. Think of it as an XML sitemap built specifically for Large Language Models.
01 — Live Code BreakdownVisiby's live llms.txt file: Annotated line-by-line
Below is the complete text of Visiby's production llms.txt file, broken down line-by-line:
# Visiby — AI Search Visibility & Citation Intelligence
> Visiby measures, tracks, and optimizes brand visibility across conversational AI search engines including ChatGPT, Perplexity, Gemini, and Google AI Overviews.
## Core Platform & Products
- [Platform Overview](https://visiby.net/ai-visibility-platform): Full breakdown of multi-engine prompt sampling, AI Share of Voice tracking, and competitor displacement monitoring.
- [Pricing & Plans](https://visiby.net/pricing): Transparent pricing details for Starter ($99/mo) and Pro ($249/mo) tiers.
## Pillar Guides & Documentation
- [What Is GEO?](https://visiby.net/blog/what-is-geo): Complete 2026 guide to Generative Engine Optimization principles and citation factors.
- [What Is AEO?](https://visiby.net/blog/what-is-aeo): Guide to Answer Engine Optimization, passage extraction, and direct answer formatting.
- [How to Rank in AI Search](https://visiby.net/blog/how-to-rank-in-ai-search): Step-by-step playbook for earning footnote citations in ChatGPT and Perplexity.
## Key Benchmark Research
- [AI Citation Benchmark](https://visiby.net/blog/ai-citation-benchmark): 172-prompt study analyzing median brand citation rates across ChatGPT (11%) and AI Overviews (4%).
- [Listicle Citation Experiment](https://visiby.net/blog/listicle-citation-experiment): Research demonstrating a 3.4x citation lift for brands included in top-10 listicles.
## Competitor Comparisons
- [Profound Alternatives](https://visiby.net/blog/profound-alternatives): Breakdown of 7 Profound AI alternatives for mid-market teams.
- [Visiby vs Otterly](https://visiby.net/compare/otterly): Direct feature and pricing comparison against Otterly AI.
- [Visiby vs Peec](https://visiby.net/compare/peec): Direct comparison against Peec AI search tracking.
Line-by-line syntax rules
- Line 1 (
# Title): Single H1 specifying brand name and product category. Keep under 70 characters so the model parses entity boundaries instantly. - Line 3 (
> Blockquote Summary): A standalone summary under 50 words defining your core business. LLMs use this blockquote as the top-level entity definition when synthesizing category overviews. - Lines 5, 9, 15, 20 (
## Section Headers): Group links logically using H2 headers (e.g. Core Products, Guides, Pricing, Comparisons). - Bullet Items (
- [Anchor](URL): Context): Provide clean Markdown link formatting followed by a 1-sentence plain text summary of what the linked resource contains.
02 — Hype vs RealityHonest assessment: Adoption is early (Hype vs Reality)
It is important to maintain realistic expectations regarding llms.txt in 2026. Automated tooling such as Chrome Lighthouse now includes audits for agentic web discovery (see Chrome Developer LLMs.txt Documentation).
[Current LLM Crawler Adoption Overview]
├── High Support: PerplexityBot, Custom GPTs, Developer RAG Agents
├── Emerging Support: Claude Web Crawler, Startup Search Bots
└── Low/Zero Direct Reliance: Google Searchbot (Relies on HTML/Schema), Bingbot
What llms.txt DOES do today
- Reduces RAG Latency: When developers or custom AI agents fetch your domain for context, parsing
/llms.txtrequires 90% fewer tokens than parsing full HTML pages. - Guides Perplexity & Custom GPTs: Perplexity and custom assistant agents frequently check
/llms.txtduring deep retrieval sweeps when available. - Prevents Hallucinations: Provides explicit entity definitions and factual pricing figures directly at the root level.
What llms.txt DOES NOT do today
- It does not replace XML sitemaps: Google Search still requires standard
sitemap.xmlfiles for indexing. - It does not guarantee instantaneous rankings: Having an
llms.txtfile will not rank a page that lacks authoritative backlinks or clear content structure.
03 — Deployment StepsStep-by-step guide to deploying your llms.txt file
Follow these 4 steps to deploy your file:
Step 1: Draft the file in Markdown
Create a plain text file named llms.txt using standard Markdown. Avoid complex HTML tags, inline styles, or JavaScript snippets inside this file.
Step 2: Place at domain root
Upload the file to your web server's public root directory so it resolves cleanly at:
https://yourdomain.com/llms.txt
Step 3: Verify HTTP 200 header response
Ensure your web server returns a standard 200 OK status with Content-Type: text/plain; charset=utf-8 or text/markdown.
Step 4: Allow AI crawlers in robots.txt
Confirm that your robots.txt file allows AI bots to access the root path:
User-agent: GPTBot
Allow: /llms.txt
User-agent: PerplexityBot
Allow: /llms.txt
04 — FAQFrequently asked questions
Want to audit how AI crawlers index your website today? Generate your free AI visibility report
Raunaq Arora is a senior software and AI engineer at Visiby, where he builds the AI-visibility measurement pipeline and dashboard. He writes about how AI-visibility tracking is measured and tooled. View full profile →