A lenient client is not a correct wire

RoutePlane served the OpenAI Responses API to Codex for months without a parse error. Then I pointed grok at the same endpoint. It failed three times, each on a required field I had never emitted.

Three required fields

created_at first. Every Responses object carries a unix-seconds timestamp. I put one on nothing: not on non-streaming bodies, not on any streamed event that embeds a response. Fixed in 058478e.

Then annotations on ResponseOutputText. The schema requires the array on every output_text part; empty is fine, absent is not. I left it out whenever there was nothing to cite. 0978048.

Then output on response.failed. This one should have been obvious. A Response always carries output, including on failure, and I dropped it on the one path where a client most needs an empty array instead of nothing. 7ac8869.

Codex parsed all three shapes for months and never said anything.

After the third miss: audit every emit site

I stopped patching individual fields and went through every place I construct a Response object: the non-streaming render, plus the streamed envelopes for response.created, response.in_progress, response.completed, response.failed, and response.incomplete. Six sites. output on failed was the only remaining gap.

One test now walks all six and requires object, id, created_at, model, status, and output as an array. That is the set I can always fill without echoing back values from the request. It lives at crates/routeplane-sdk/src/language_model/protocol/tests.rs.

I do not emit output_text. The SDKs derive it from the output items; it is not a required wire key, and adding it because a client might want it is the same guessing that got me here.

Nothing in my suite had ever checked the shape of an emitted Response. That is why an external client found three of these one at a time instead of me finding all three at once.

Two things I am not fixing. grok makes a session_title side call under its own product model id grok-build, which I cannot route, so every grok run leaves one local_reject in the log. Adding a routing alias would clean the log by claiming a model I do not have. It stays noisy. And a grok run that answers correctly proves nothing on its own. A warm ~/.grok/models_cache.json can carry a hardcoded production base_url and answer from xAI without touching RoutePlane at all. The only proof is a new insight record.

Four other green signals from the same week

Three tests in the suite passed while production was free to violate what they claimed. One asserted a legacy providers JSON by key presence, so models, active, and api_base could hold wrong values. One human-safety check ran over a fixture whose providers list was empty, so its leak loop never executed and every assert was free. One “allowlist” for a downstream catalog export was a substring match against known-bad names, so any field with a new name passed. They got real assertions in d479524.

SpawnAgent::Codex and SpawnAgent::Kimi shipped at 1,659 passing tests with five green gates and could not complete a single turn against a live daemon. Every spawn test asserted argv and env construction; none asserted that the model id I emitted was routable. Kimi asked for a model literally named routeplane, because the alias constant was also the upstream model field. Codex sent its own default.

My own end-to-end check passed on a run that made no request. It prompted reply exactly: SPAWN_E2E_OK and grepped the transcript for SPAWN_E2E_OK, which the harness writes there itself when it echoes the prompt. Now it asks what is 2+2? answer with only the digit and expects 4.

And on 2026-07-30 the installed ~/.cargo/bin/routeplane was still the Jul 25 binary, 126 commits behind main, while routeplane status reported healthy the whole time. :4356 returned 404 for /v1/settings, which is a route only the newer code serves. Every end-to-end proof that week had run against scratch daemons built from branches. Nobody had restarted the one actually serving traffic in four days.

What I do now

Keep the field-by-emit-site audit and the pin test. Treat a grok parse failure as a bug in my Responses output until proven otherwise. A green suite does not mean the daemon can serve a turn. Check for a new insight record with result: ok. An end-to-end prompt has to demand an answer the harness cannot produce by echoing. A review lane that cannot read ~/.routeplane or see $HOME should say so rather than report a record it did not see. And a test nobody has watched fail is not a test: mutate the line it protects, watch it go red, revert.

Codex was a fine client to develop against and a bad specification to develop from.