DemoBitesTechnical GEO has become a magnet for magic file thinking.
Add this directive. Publish that manifest. Install one schema block. Suddenly every AI system understands your site.
Reality is less exciting and more useful.
There are mature foundations, emerging conventions, and experiments. They should not be treated as equally proven.
robots.txt: access policy
robots.txt is one of the oldest control surfaces the web offers to machines.
AI has made this more complicated, because “AI crawler” can mean
- a crawler used for search,
- a crawler used for retrieving answers in real time,
- a crawler used for model training,
- a fetch triggered by a user,
- an autonomous browser agent.
Those are not the same business use case.
OpenAI documents OAI-SearchBot for search discovery and recommends allowing it if publishers want content eligible for ChatGPT search discovery and citations.
Anthropic documents three separate crawlers, ClaudeBot for training data collection, Claude-User for fetches triggered by user requests, and Claude-SearchBot for search, each with its own robots.txt user agent.
Cloudflare now groups AI crawler traffic into search, agent, and training categories, and layers content use signals on top through its managed robots.txt.
The lesson is simple.
Decide what you want machines to do before copying someone else's robots.txt.
Sitemaps: still boring, still useful
Sitemaps help discovery systems locate important URLs.
They are especially useful when a product site contains a growing library of feature pages, release pages, integrations, documentation, and localized variants.
A sitemap does not make weak pages authoritative.
It helps systems know the pages exist.
Structured data: describe reality
Structured data can help machines interpret page meaning.
Useful standard types may include
- Article
- BreadcrumbList
- Organization
- Person
- VideoObject
- Product, in appropriate contexts
The rule is short.
Mark up what is actually on the page.
Do not invent “AI schema” because a blog post promises more citations. Google's guidance for generative AI in search does not require any special AI schema.
Q&A markup: content and schema are different
Clear Q&A is excellent content.
That does not mean every Q&A section should receive QAPage structured data.
Google's QAPage documentation is intended for pages centered on one question with answers submitted by users. Ordinary FAQs written by the company itself do not qualify.
Google also deprecated FAQ rich results in 2026.
Keep writing useful questions.
Stop treating schema as the reason to write them.
llms.txt: interesting, emerging, not magic
llms.txt is a community proposal designed to give LLMs and agents a concise map to your most useful machine readable content.
The proposal evolved in 2026 with additional conventions for discovering Markdown alternatives and the relevant llms.txt file.
It has increasing adoption among documentation platforms and tools.
That makes it worth evaluating.
But llms.txt does not create knowledge that is missing from the site.
A perfect file pointing to vague pages still points to vague pages.
What it is not
A ranking cheat code that lifts weak pages into AI answers.
What it is
A useful navigation layer that points agents to your best machine readable content.
Markdown alternatives
HTML often contains navigation, UI chrome, scripts, and components agents may not need.
Some documentation platforms expose clean Markdown versions of pages.
For documentation heavy sites, machine readable representations can reduce extraction noise.
For a commercial site, implement them carefully so there remains one clear canonical truth.
The maturity stack
Not everything labeled technical GEO deserves the same confidence. Sorting the advice by maturity keeps the effort honest.
Established foundations
Emerging, increasingly practical
Treat with skepticism
Technical GEO is important.
It is just not a substitute for product truth.
Questions people ask
- Do we need llms.txt today?
- Not universally. It is an increasingly adopted proposal worth testing, especially for documentation heavy sites.
- Should we allow AI training crawlers if we want AI search citations?
- Those are separate policy questions. Search discovery, answer retrieval, and model training are handled by different crawlers with different controls, so review each provider's current documentation before deciding.
- Is there special GEO schema?
- No generally accepted special schema is required. Standard structured data that describes what is actually on the page remains the sound approach.
- Are sitemaps still useful?
- Yes. Discovery of canonical URLs remains foundational. A sitemap does not make weak pages authoritative, it simply helps systems know the pages exist.