AEO Bench

Atom

Can your website be read by an AI agent, and do the techniques that promise to help actually work? AEO Bench is our open, pre-registered benchmark for agent readiness and answer-engine optimization: controlled site fixtures with real HTTP semantics, seeded questions with known answers, mechanical grading, and token cost as a first-class outcome.

Study 1 measured the retrieval-class techniques from Cloudflare's agent-readiness proposal (llms.txt, sitemap.xml, and markdown content negotiation) on a well-linked site; Study 2 rebuilt the site so discovery files were the only path to the answers; Study 3 measured how an agent harness should consume a site index; Study 4 measured the capability layer (MCP server cards and structured endpoints). Together: 2,808 scored agent runs across six models. Hypotheses, corpora, and graders are committed before any scored run; results are published as found, corrections included.

Methods & Provenance

Four pre-registered studies, 2,808 scored agent runs across six models at temperature 0 (July 2026). Every study is registered by commit before its first scored run (hypotheses, task corpora, site fixtures, and graders all committed first) and published as found, corrections included: Study 1′ re-scored the frozen records under a corrected grader, and both readings are public. The fixture is Petrel & Pine Supply Co., a deterministic, fictional retail site served in-process: 40 well-linked pages in Study 1; a ~300-page, weakly-linked catalog with twelve registered orphan pages in Studies 2 and 3; and a structured product endpoint, order-status API, and MCP server card for Study 4's capability layer. Agents reach the site only through a fetch tool whose description never hints at any technique, so every discovery is the model's own; from Study 2 onward a structured submit_answer channel replaced free-text answers, closing Study 1's grading lessons mechanically.

0/900

discovery-file consultations across Study 1's 900 agent runs; with llms.txt and sitemap.xml present and advertised, agents entered at the root and followed visible links

28/28

present-class facts found by every major model in every arm, including bare HTML with no aids at all; no retrieval aid can improve a ceiling (Study 1)

74% vs 20%

markdown negotiation's input savings for a wasteful agent (gemini) vs a frugal one (opus); the saving is proportional to the waste (Study 1)

0/10

orphan-page tasks solved in every file-bearing arm, every model, on a 300-page site built so discovery files were the only path to the answers (Study 2)

148/150

unreachable facts confidently declared not-on-site through a structured answer channel; unlinked content is not just unfound, it is denied (Study 2)

0→10/10

the orphan class rescued by ONE sentence of fetch-tool affordance; path-guessing collapsed (kimi: 43 guessed 404s to zero) (Study 2)

9.4k vs 68.9k

input tokens per solved task on Haiku 4.5: curated site index prefetched into context vs unaided; 4.4× cheaper than the hint at equal or better accuracy (Study 3)

2.68×

what the giant everything-index costs relative to the curated slice, for zero added accuracy; a token tax, not a behavior failure (Study 3)

MCP server cards consulted unprompted, by any model: the third well-known file class to score a universal zero, after llms.txt and sitemap.xml (Study 4)

0/24

product questions answered via a fully-disclosed but unmounted API; knowing an endpoint exists changes nothing, holding it as a tool changes everything (Study 4)

5.4×

input-cost cut when endpoints are mounted as harness tools (haiku: 51.4k → 9.5k mean input tokens per cell); mounted was the cheapest arm for every model (Study 4)

invented shipping statuses in 288 cells with four fake order numbers per arm; every nonexistent order was reported not found (Study 4)

The study series · four studies + a registered re-score

Every Study, One Page Each

Every study is pre-registered before its first scored run and gets its own page with charts, tables, and a verdict. Study 1′, the registered corrected-grader re-score, is reported alongside Study 1. Grouped by theme:

Retrieval-class techniques

Study 1 · Nobody reads llms.txt: the aids don't change what agents find; one changes what it costs Across 900 agent runs on a fictional 40-page site, no model consulted llms.txt or sitemap.xml even once; correctness saturated in every arm including bare HTML; markdown negotiation was discovered unprompted by some models and not others (opus 64%, sonnet 0%), and its token savings scale with how wasteful the agent was to begin with (gemini saves 74%, opus 20%).
Study 2 · The discovery-file mechanism: nothing reads llms.txt even when it is the only way, and one sentence fixes everything On a 300-page weakly-linked site with orphan pages reachable only through discovery files, every model scored 0/10 on orphan tasks in every file-bearing arm because nothing reads the files (opus consulted them 0 times in 128 chances); one sentence of tool-description affordance took orphans to 8 to 10 of 10 everywhere protocol-compliant, and unreachable facts were confidently declared not-on-site 148 of 150 times.

Agent harness design

Study 3 · The affordance study: the index goes in the context The series' first gate PASS: prefetching a curated site index into the agent's context matches or beats every other design at a fraction of the cost (Haiku: 9.4k input tokens per solved task vs 68.9k unaided and 41.3k under the hint), every deliberate affordance dissolves Study 2's orphan catastrophe completely, and the giant everything-index costs 2.68 times the curated slice for zero added accuracy.

The capability layer

Study 4 · The capability layer: three files nobody reads, one sentence that replaces them, and the tools that pay for themselves MCP server cards join llms.txt and sitemap.xml as the third well-known file class with a universal zero: no model consulted one unprompted. But one affordance sentence buys the whole card-to-API chain at every tier, mounted tools flip API habits and cut costs up to 5.4 times, and zero invented order statuses appeared in 288 cells.

The article series · every post, in order

The Full Series

Every post in the AEO Bench series, in order. Start at the top for the whole story, or jump straight to the latest study writeup.

Introducing AEO Bench: Measuring Agent Readiness Instead of Guessing Jul 30, 2026
Nobody Reads llms.txt (Yet) Jul 30, 2026
One Sentence Beats Every File Jul 30, 2026
The Index Goes in the Context Jul 30, 2026
Three Files Nobody Reads Jul 30, 2026