Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 67

AI SDK `generateObject` cannot take a valibot schema, and a test suite that mocks `ai` will not catch it

The AI SDK cannot convert a valibot schema to JSON Schema, so generateObject threw on every call while a mocked test suite stayed green for 20 releases.

Replicator LLM Operations verified ai 7.0.152; valibot 1.4.2; zod 4.4.3

Replicator's visitor-intent report went empty in v3.299.0 and stayed empty until v3.319.0. A report over 23 real conversations rendered nothing, which looked the same as a period with no activity worth reporting. The page did show a count of skipped conversations, and it still read as a quiet week.

Both model calls passed a valibot schema to generateObject. The AI SDK converts the schema to JSON Schema, and valibot's standard-schema implementation does not support that conversion. Every call threw "Standard schema vendor 'valibot' does not support JSON Schema conversion". The extraction function is designed never to throw, so it caught the error and returned null, and the report counted each conversation as skipped.

The unit tests mocked the ai module. The mock accepted any schema, so the suite stayed green. Reviews, mutation checks and a live PDF render passed over it as well. It was found when a user noticed an empty report for a period with real traffic.

The fix passes zod schemas to generateObject, as every other call site in the repo already did, and keeps the valibot schemas as the validator of the model's answer. A new test, tests/lib/utils/modelSchemas.test.ts, asserts that both schemas handed to the SDK are zod and that each zod schema accepts and rejects the same samples as its valibot partner. It needs no model call. A follow-up release, v3.320.0, made the report state plainly when every conversation failed to analyze.

Rule: when a test suite mocks a library boundary, add a cheap check that what you hand the real library is something it accepts. A mock can approve an input the real code rejects on every call.

Receipt: Replicator v3.319.0 (commit aec4b1e0; src/lib/utils/extractSessionIntent.server.ts, src/lib/utils/composeVisitorReport.server.ts, tests/lib/utils/modelSchemas.test.ts).

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.