AI Agents Are Already Judging Your Docs
Google Cloud AI Engineering Director Addy Osmani formally introduced **Agentic Engine Optimization **(AEO) in April 2024—a set of concrete practices for optimizing product documentation to be Agent-friendly. This is not a future concept but a current reality: when engineers use AI coding agents like Cursor, Claude Code, Windsurf, and Gemini CLI, whether a product gets picked up—or silently rejected—hinges on how well its documentation supports Agent consumption.
AEO emerged from a hidden but critical data paradox: most AI Agent traffic appears in traditional analytics as “low-quality” visitors with 100% bounce rate—zero clicks, zero time on page—while in reality, Agents made an irreversible judgment and moved on. The silent “death sentence” occurs when the first 500 tokens fail to answer three questions: What is this? What can it do? How do I get started?
Three Counterintuitive Differences in Agent Reading Behavior
AI Agents consume documentation in ways fundamentally unlike humans:
- Quantified patience: Agents decide within 400 milliseconds; the first 500 tokens must contain core answers. No tolerating lengthy intros.
- Strict token budget: Quick start guides should be <15,000 tokens, API reference pages <25,000, conceptual docs <20,000. Cisco’s firewall quick-start guide reaches 193,217 tokens—enough to fill or overflow most agents’ context windows, causing truncation, skipping, or hallucinated answers.
- UI blindness: Sidebars, breadcrumbs, footers, and interactive sandboxes are pure noise. HTML pages consume far more tokens than Markdown due to wrapper divs, CSS classes, ARIA attributes, and inline styles.
Interestingly, unlike human developers (4–8 minutes, navigation, multiple clicks, code trials), agents complete full assessment in a single GET request within 400ms. Research referenced by Osmani confirms this behavior across nine major coding agents.
The Six-Layer AEO Framework
Osmani’s implementation roadmap ranks effort against impact:
| Layer | Optimization Action | Estimated Time | Key Benefit |
|---|---|---|---|
| 1 | Audit robots.txt | 10 minutes | Avoid blocking Anthropic/OpenAI/Google user agents |
| 2 | Publish llms.txt sitemap | Hours | Markdown directory with page titles, summaries, and token counts |
| 3 | Write skill.md capability statement | Medium | List features, required params, rate limits for fast suitability check |
| 4 | Enable .md URL access | Simple | Plain Markdown avoids HTML overhead and JS dependence |
| 5 | Expose token counts in metadata | Simple | Meta tags or headers let agents decide before loading |
| 6 | Add Copy for AI button | Low | One-click clean Markdown extraction |
The first three layers—robots.txt audit, llms.txt, and token metadata—can be completed over a single weekend.
Who Should Care Now?
- Adopt now if: You build APIs, SDKs, or developer-facing SaaS; your docs team wants higher agent-indexing rates; your team has basic engineering resources (夕露 is sufficient for layers 1–3).
- Wait if: Your product is strictly local/offline with no API/coding integration; your documentation is already static Markdown without client-side rendering;
writen in the end
AEO mirrors SEO in structure—optimize for the crawler, not the human—but with zero tolerance. Osmani notes the irony: many AEO optimizations align with good human documentation practices—frontloaded answers, focused single-page scope, clean hierarchy, noise reduction. Humans can skip, scroll, search; agents get one pass. The quietly unfolding shift is already filtering products by developer-centricity, not brand or marketing.