# GeoXylia
> Technical GEO hub: allow the right AI retrieval bots, treat schema as infrastructure, skip llms.txt as a lever. Guides for AI visibility live here.
← Blog/Technical GEO/Technical GEO: AI Crawlers, Schema, and the Infrastructure That Makes You CitableTopic HubTechnical GEOIntermediate

## Technical GEO: AI Crawlers, Schema, and the Infrastructure That Makes You Citable

The hub for GeoXylia's technical GEO cluster: which AI crawlers read your site, why schema is infrastructure not a lever, and the honest truth about llms.txt.

Ethan Lim2026-08-0112 min

Last updated: 2026-08-15

Share:

Technical GEO is about being reachable and readable for AI retrieval systems: 122 of 600 audited sites are partly invisible because bot protection blocks AI crawlers, schema shows no measurable citation effect, and llms.txt files go unread 97% of the time (Ahrefs, 2026).

This hub collects GeoXylia's technical GEO cluster: AI crawler governance, schema markup, llms.txt, rendering, and the other infrastructure questions that decide whether AI systems can read your site at all. The theme across the cluster is honesty about what is a lever and what is infrastructure. Two once-hyped tactics, llms.txt and schema-as-citation-magic, fail the evidence test. One thing does not: if AI crawlers cannot reach your page, nothing else you do matters.

Who is this for? Engineers, technical SEOs, and the marketers who translate their work. The distinction that runs through every guide here is retrieval versus training: retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Google-Extended) fetch pages to answer live questions, while training bots (GPTBot, ClaudeBot, CCBot) collect data for model training. You can block the second group freely; blocking the first costs you citations.

## The crawler governance model

BotRolePolicyOAI-SearchBotChatGPT live retrievalAllowChatGPT-UserChatGPT web browsingAllowPerplexityBotPerplexity retrievalAllowClaude-SearchBot / Claude-UserClaude retrievalAllowGoogle-ExtendedGoogle AI surfacesAllowGPTBotOpenAI trainingOptional to blockClaudeBot / CCBotTrainingOptional to blockMeta-ExternalAgentTrainingOptional to block

The rule of thumb: never block retrieval bots — they decide the citations users see today. A published test found citations dropped 85% (published crawler test, 2026) after a site blocked retrieval bots. Training bots are a policy choice you can make freely. Audit the allowance list every quarter, because new bots appear as platforms ship retrieval features.

## Is llms.txt worth my time?

Keep expectations low. Ahrefs found 97% of llms.txt files are never read by AI crawlers, adoption sits at 10.13% of domains with no meaningful traffic-tier difference (SE Ranking, 2026), and no major AI platform officially supports the format. The honest framing: llms.txt is infrastructure, not a lever. It costs almost nothing to publish and it functions as a content map for AI coding assistants and developer tools, which is a real but narrow audience. It is not crawl control like robots.txt, and it will not move your AI search citations. If you publish one, keep it accurate and short, and spend your optimization effort where the evidence is: extractable content, entity signals, and crawler access. The guides in this cluster show exactly how to do each of those.

## Does schema markup get me more AI citations?

No measurable effect on already-cited pages. Ahrefs ran a 1,885-page quasi-experiment adding JSON-LD schema to pages and measured AI Overviews at -4.6%, AI Mode at +2.4%, and ChatGPT at +2.2% (Ahrefs, 2026), all within noise. Studies that claim schema lifts citations are observational, and the correlation runs the other way: well-optimized sites tend to have both schema and citations. The correct frame is that schema is necessary infrastructure. Organization with a full sameAs array, Article with author and dateModified, and BreadcrumbList give AI systems the machine-readable context they need to recognize your entity and your content's provenance. One nuance remains: schema may help pages with no citations at all cross an eligibility threshold, but that effect is plausible, not proven, while on already-cited pages the measured effect is noise. Keep schema valid, complete, and matching what users see. Just do not expect it to produce citations by itself; visible, answer-shaped text is what AI extracts.

## The rendering requirement

AI crawlers do not execute JavaScript. Content that only exists after scripts run is invisible to every retrieval system, and JavaScript-injected schema markup is never read. The baseline is server-side or static rendering: the answer text must be in the initial HTML response. Single-page applications that hydrate client-side are the most common technical cause of AI invisibility — the page looks fine in a browser and does not exist to a crawler. Verify with a simple test: view-source your page and confirm the answer text and schema appear in the raw HTML, not only after scripts execute.

## The robots.txt baseline

A correct allowance block takes ten minutes and removes the most co
## Links
- [GXGeoXylia](/)
- [Features](/features)
- [Pricing](/pricing)
- [Methodology](/methodology)
- [Blog](/blog)
- [FAQ](/faq)
- [About](/about)
- [Free Audit](/audit)
- [Technical GEO](/blog/technical-geo)
- [476-site audit](/blog/state-of-geo-2026-benchmark-476-sites)
- [Ahrefs: schema quasi-experiment](https://ahrefs.com/blog/)
- [SE Ranking: llms.txt analysis](https://seranking.com/blog/llms-txt/)
- [llmstxt.org](https://llmstxt.org/)
- [Google Search Central: robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots-txt)
- [llms.txt Examples: 10 Annotated Implementations You Can CopyAdoption of llms.txt](/blog/llms-txt-examples-10-real-implementations)
- [llms.txt: The Complete Technical Setup Guide for 2026Here's the complete ll](/blog/llms-txt-the-complete-technical-setup-guide)
- [llms.txt and Content Strategy: Making AI-Ready Content for 2026As AI answer engi](/blog/llms-txt-and-content-strategy-making-ai-ready-content)
- [Technical SEO for AI Search: The 2026 Complete GuideTraditional technical SEO do](/blog/technical-seo-ai-search-2026)
- [Image SEO for AI Visual Discovery in 2026AI search is going multimodal. Learn ho](/blog/image-seo-ai-visual-discovery)
- [GEO Fundamentals: What GEO Is, Why It Matters, and How to Get Started12 min](/blog/geo-fundamentals)
---
Generated by [GeoXylia](https://geoxylia.com) — AI Visibility Platform