I've been working on LLM extraction in Spring Boot services (invoices, forms, that kind of structured data), and kept hitting the same issue.
Spring AI 2.0's `validateSchema()` is great: it checks the response against the record's JSON schema and retries with the error if it fails. But the bugs that actually hurt me passed that check:
- A line item silently dropped, so the rows no longer added up to the total
- A value that never appeared anywhere in the source document
- Correct output on Monday, slightly different output after a prompt tweak on Friday
The JSON was valid and deserialized cleanly into my record every time. The data was just wrong.
Using a second LLM as a judge in tests felt like the wrong fix: slower CI, token costs, and tests that don't give the same verdict twice.
So I started writing deterministic checks as plain JUnit tests instead:
- Cross-field arithmetic with BigDecimal (parts must sum to totals)
- Provenance: extracted values must actually appear in the source
- Latency and cost budgets per call
- A golden set of JSONL examples that runs on every PR, so a prompt or model change shows up as a failing build
I ended up packaging this as an open-source library, Outputgate (on Maven Central, works alongside Spring AI rather than replacing anything). One known limitation: provenance matching is currently verbatim, so `1,200.00` vs `1200` can fail correct output. Normalized matching is what I'm working on next.
Repo: https://github.com/ravismash/outputgate
But I'm mostly curious how others handle this:
Do you test LLM output correctness in CI at all, or only at runtime?
Are you using LLM-as-judge evaluators, hand-written checks, or something else?
What's the worst "valid but wrong" output you've caught?