We Built Our Chat Agent the Way Our Research Told Us To

There is a chat button on every page of this site now. It is called Ask Eljay, and if you ask it what Lightning Jar does, it will answer from the site's own published content and link its sources. That part is table stakes in 2026. The part worth writing about is that nearly every design decision inside it was settled by a pre-registered experiment before we wrote the production code, and the numbers that settled each decision are public.

This is the story of what that looks like in practice: an agent built almost entirely out of findings from AEO Bench, our open benchmark series on agent readiness, applied to ourselves.

The retrieval design came from a measurement, not a debate

The core question for any site agent is how it finds things. Give it search tools and hope? Hand it a map? Let it crawl?

We ran that as an experiment before deciding. Study 3 measured four harness designs on a frozen 300-page fixture: an unaided agent, a hint pointing at the site's index files, a dedicated read-the-index tool, and a curated index placed directly in the system prompt. Every deliberate design solved the hard retrieval class completely. The difference was cost. On the model we planned to ship, input tokens per solved task fell from 68.9k unaided to 41.3k under the hint, 13.5k with the index tool, and 9.4k with the curated index in context.

So Ask Eljay carries a curated index of this site in its system prompt: every section, every research study with its slug, every package, and pointers to search for the long collections. Not everything, deliberately. The same study measured a giant everything-index at 2.68 times the cost of the curated slice for zero added accuracy.

The index rides in a static, cacheable system block, which matters because Anthropic's prompt caching serves repeated static content at roughly a tenth of the input price. And when the agent reads pages, it reads markdown, not HTML: Study 1 measured markdown saving between 20 and 74 percent of the tokens HTML costs an agent, depending on how wasteful the agent was to begin with.

The model was chosen by a registered gate

Ask Eljay runs on Haiku 4.5, the cheapest model tier we measured. That sounds like a budget decision. It was actually a gate.

We added Haiku to the benchmark by registered backfill, specifically as the candidate for this agent, and let the studies judge it. Its profile came back stronger than models costing several times more: a perfect score on every present-fact retrieval class in Study 1, the only clean sweep of the linked classes in Study 2, zero invented facts across every cell it touched, and the best economics in Study 3's readiness gate. We did not pick the cheap model and hope. We registered the bar, and the cheap model cleared it.

The grounding rules encode our own failure data

Study 2 produced the finding we think about most: when content was unreachable, agents did not say they could not find it. They authoritatively declared it did not exist, 148 times out of 150. An agent that answers "we don't do that" when the real answer is "I didn't look hard enough" is quietly poisonous for a business site.

So Ask Eljay's system prompt makes the claim expensive. Before it may say the site does not cover something, it must have actually searched this turn. Only after tools come back empty may it say no, and then it points to a human instead.

The same study measured how cheap the fix for missed affordances can be. One sentence added to a fetch tool's description took the hardest retrieval class from zero out of ten to ten out of ten. We got to apply that lesson to ourselves within a day of turning on chat logging: the logs showed the agent burning four of its six tool steps guessing wrong formats for a study lookup, because our own index listed studies as paths and the tool wanted bare slugs. The fix was the study's fix. Accept the formats agents actually guess, and make the error message teach the format when they still miss.

The defenses came from production evidence, not vendor defaults

A public chat endpoint spends real money on every request, so the abuse question matters. Here we had evidence from a separate production project: when we instrumented an automated client that was quietly responsible for most of a third-party API bill, the thing that eliminated it was not bot classification. It was requiring a human-origin gesture before doing anything expensive. The automated traffic simply never produced one. A heavyweight bot-detection challenge we trialed on that project cost real mobile users around twenty seconds of latency and got reverted the same morning.

Ask Eljay inherits that posture. The chat SDK does not even load until a person touches, clicks into, or focuses the chat. Behind that sit rate limits in Redis shared across server instances, size caps on everything, and a fail-closed core. What we deliberately did not add: proof-of-work bot challenges. The evidence said they tax humans more than they stop abuse for a surface like this.

One more economy: the suggested starter questions are fixed strings, so their answers are cached. A cached answer returns in about seventy milliseconds instead of several seconds of model streaming, and costs nothing.

The logs closed the loop

Every conversation ships, with contact details scrubbed, to a review surface where we read what actually happened. That pipeline paid for itself on day one with the slug-guessing discovery above, and it is how the next round of decisions will get made: which starter questions people actually click, whether voice input earns its keep, where the model's six-step budget runs out.

That is the real shape of the method. Measure, ship the winning design, instrument, and let the instrument tell you what to measure next.

What we would tell you to take from this

Not the specific numbers. Ours are true for the models we tested in mid-2026, on our content, and we would be the first to say this discipline is not settled science. What transfers is the order of operations: the experiment came before the architecture, the architecture cites the experiment, and the production logs feed the next experiment.

Ask Eljay is the demo. The research is public, the playbook distills it, and the agent answering questions in the corner of this page is what it looks like when you actually follow your own advice. Go ask it something.