Codex upstreams can silently serve a different model than the one requested
(HTTP 200, with response.model naming the substitute). The proxy kept no record
of it: nothing logged, nothing reported, only the pass-through response body.
- Add Record.ResponseModel to sdk/cliproxy/usage, aligned with the existing
ResponseServiceTier field, and emit it from the redis usage queue as the
optional response_model payload field alongside response_service_tier. Only
the record for the requested model carries it: additional-model records
(image generation tool usage) describe a side model the upstream response
never refers to, and would otherwise look like a substitution downstream.
- Add internal/runtime/executor/helps/response_model.go with
extractCodexResponseModelEvent (SSE frames and raw JSON, restricted to the events
that embed the authoritative response object, rejecting non-string and
oversized upstream model names) and IsCodexModelSubstituted (both sides
trimmed, lower-cased and stripped of thinking suffixes, dated aliases such as
gpt-5.6-terra-2026-05-13 accepted in either direction).
- UsageReporter records the served model on the event path and emits the WARN
when the attempt publishes its usage record, so no logging work happens
before the first event is forwarded. Repeats are throttled per
(auth id, requested model, served model) with a 10 minute window, because on
an affected credential every request is substituted and an unthrottled
warning would mirror the whole request volume into the logs. The credential
is labelled auth_index=<index> only: codex credential file names embed the
account e-mail, which must not be written to the logs at request rate.
Coverage, by entry point. The served model is observed on the HTTP streaming
path (both the bootstrap-buffered handshake and the streaming goroutine), the
HTTP non-streaming Execute loop, the websocket streaming and non-streaming
paths, and the two /responses-shaped image entry points. The remaining codex
entry points cannot report it and are therefore left alone: executeCompact
(/responses/compact answers with a compaction object that has no event type and
no response.model), the two direct image endpoints (/images/generations and
/images/edits answer in the Images API shape and stream image_generation.*
events), and CountTokens (counts locally with tiktoken, never reaching an
upstream).
TokenAccountingSchemaVersion is not bumped: it versions the token accounting
contract (token breakdown semantics), and this change only adds an optional
non-token field that leaves existing consumers and all token math untouched.
Note: response_model ships with the usage record and is the counting source;
the WARN is a throttled alerting signal and must not be used to count
substitutions.
Tests: table-driven unit tests for both helpers over real model ids, reporter
tests covering the published record, the single throttled warning, the absence
of account identifiers in it, concurrent observation and publishing under
-race, the throttle window and its entry bound, an executor-level guard for the
observeCodexTokenEvent wiring and the per-model records, plus a redisqueue
payload assertion for response_model. gofmt, go vet, go test -race on the
touched packages and go test ./... are clean.
- Add `stream-bootstrap-timeout` configuration (defaulting to 0/unlimited, recommended 20s behind reverse proxies) to bound how long early handshake or trickled events may hold response headers.
- Release stream buffering into normal in-stream delivery once the time budget is exhausted on both SSE and WebSocket executors.
- Deliver post-timeout overload and status-bearing errors in-stream rather than triggering credential failover, preventing latency doubling on long reasoning turns.
- Provide thread-safe mock clock test harness and comprehensive unit tests covering timeout release, unlimited default, disabled ceilings, and post-timeout error delivery.
The bootstrap buffer released the stream on the first frame outside the
handshake allow-list. Upstream sends `keepalive` heartbeats and
`response.output_item.added` before the first token, so the downstream headers
were committed while nothing had been produced, and an overload rejection
arriving afterwards could no longer be retried on another credential.
Extend the allow-list with `keepalive`, and with `response.output_item.added`,
`response.content_part.added` and `response.reasoning_summary_part.added` when
they announce something that is still empty. `response.output_item.added` is
accepted only for `message`, `reasoning`, `function_call` and `custom_tool_call`
items, and only while their content, summary, arguments or input is empty: every
other item type stands for a server-side operation that may already be running -
a `web_search_call` is announced with status "in_progress" and its `searching`
event follows immediately - and failing the attempt over after one would run
that operation again on another credential. Part announcements are matched on a
closed list of textual part types for the same reason.
The list stays closed. "Nothing has happened yet" cannot be derived from the
absence of a TTFT token, because TTFT deliberately ignores server-side tool
traffic such as `response.shell_call_output_content.delta` and its `.done`
counterpart, so an unrecognised frame, an unrecognised item type and an
unrecognised part type all release the stream exactly as before.
Holding those frames also required fixing the bound on how much may be held.
codexBootstrapMaxBufferedEvents was enforced with len(bufferedChunks), and a
chunk count bounds only the downstream formats that render every upstream frame:
the OpenAI Chat Completions and Gemini translators return zero chunks for a frame
they do not recognise, which is true of every frame added here. Count frames read
from the upstream instead, and cap what a bootstrap accumulates over the upstream
frames and the chunks they translate into. The SSE scanner walks physical lines
and an event can arrive as one line (": keepalive"), two, or three
(event:/data:/blank), so the frame budget is sized for the widest framing rather
than derived from any separator. The websocket loop counts a message as soon as
it is read, before the branches that skip non-text and whitespace-only messages,
so a peer sending only frames the loop skips cannot hold the downstream headers
open; the message that exhausts the budget is still processed and delivered
rather than dropped. Both caps are checked before a frame is admitted, so one
oversized frame cannot be taken on the strength of an empty buffer. This widens
the window where one frame rendered one chunk - the websocket path effectively
moves from 16 frames to 48.
An empty `response.incomplete` seen while buffering is now delivered in-stream
instead of failing the attempt over, matching the websocket executor and the
documented contract that only overload and rate-limit rejections trigger
failover.
`isCodexHandshakeMetadataEvent` is renamed to `isCodexBootstrapBufferableEvent`
because the list is no longer only handshake metadata. Both config doc sites
describe what is held, that heartbeats are held too, that SSE also holds the
lines around a `data:` frame, and that the hold is bounded by frames and bytes
rather than by wall-clock time.
- Map upstream `model_not_found` errors to HTTP 404 before evaluating generic invalid request types in Codex terminal error handling.
- Prevent treating structured model not found responses as client request faults to preserve credential rotation.
- Recognize model access denial errors to apply model-level cooldown and failover.
- Respect `disable_cooling` configuration during model-level cooldown processing.
Closes: #5635
- Broaden pattern matching for Codex model capacity errors.
- Classify model capacity rejections as overload bootstrap failures to enable failover.
Closes: #5634
- Format streaming error payloads with nested error objects matching official OpenAI Responses SSE specifications.
- Extract and propagate sequence numbers from upstream terminal events and framer states.
- Use `json.Number` to prevent precision loss for large integers and token metrics.
- Sanitize sensitive keys recursively across nested error objects without dropping custom fields.
- Add `model-level-cooling` configuration option to Codex settings.
- Scope `usage_limit_reached` quota cooldowns to the requested model instead of the entire credential when enabled.
- Propagate model-level cooling checks across HTTP, SSE, and WebSocket execution paths.
Closes: #5619
- Check `error.message` and `message` for retry advice on upstream `server_error` responses.
- Treat server errors indicating the request can be retried as eligible overload bootstrap failures.
- Normalize complex constant `oneOf` and `anyOf` tool parameter schemas into equivalent enums to prevent upstream aborts.
- Escape property keys containing dots and colons during schema updates to prevent invalid path splitting.
- Identify terminal `response.incomplete` events with zero output tokens and no content as upstream failures.
Closes: #5551
- Mark Codex usage limit errors as credential-scoped across HTTP and WebSocket executors.
- Support both top-level and nested error structures with case-insensitive matching when parsing retry-after resets.
- Propagate prevalidated candidate context to session affinity and built-in selectors during auth selection.
Closes: #5529
* feat(codex): add opt-in stream bootstrap buffering
The upstream smuggles capacity rejections into an HTTP 200 stream. The
handshake events arrive normally and only a later event carries
{"error":{"type":"service_unavailable_error","code":
"server_is_overloaded"}}. By then the executor has already handed the
first chunk downstream, the response is committed, and the conductor can
no longer retry on another credential, so the request fails even though
other credentials were available.
When codex.stream-bootstrap-buffering is enabled the executor holds back
the handshake events until it can tell whether the stream carries real
output or a rejection. An overload rejection then fails the attempt
before any chunk is delivered, letting the conductor retry on another
credential; every other terminal failure is flushed in order and
delivered in-stream exactly as before.
Detection uses an event-type allow-list rather than a fixed count. On the
websocket transport codex.rate_limits and codex.response.metadata arrive
before response.created, making the first generated event the fifth
frame, so a small counter would release the stream before the rejection
is visible. Buffering is bounded and hitting the bound degrades to the
original unbuffered behaviour.
Two details are load-bearing. The error must be returned synchronously:
delivering it as the first stream chunk makes ExecuteStream downgrade it
into a committed 200 and the status is lost. And the websocket path must
not signal an upstream disconnect for a rejection it intends to retry,
because the downstream handler closes the client connection on that
signal and the retry would have nowhere to deliver.
The 503 status is produced only on this path rather than in the shared
codexTerminalFailureStatus mapping, so disabling the feature restores the
previous behaviour exactly, including cooldown classification and
retry-after parsing.
Defaults to false: response headers are withheld until generation
starts, which can trip client or reverse-proxy read timeouts.
* test(codex): pin bootstrap overload failover through the conductor
Executor-level tests cannot show what the client finally receives. These
exercise ExecuteStream end to end to pin three properties that are easy
to regress:
- consecutive overloaded credentials are skipped until one serves the
request, and retries are capped by max-retry-credentials rather than
multiplying with request-retry
- exhausting the pool surfaces the upstream status instead of a
committed 200 stream
- with buffering disabled the rejection stays an in-stream error on a
committed stream, which is the behaviour the feature must preserve
The third case also documents why the executor returns its error
synchronously: an error arriving as the first stream chunk is wrapped and
downgraded into a committed 200, silently losing the status.
Forward an upstream failure to Responses websocket and SSE clients only when the request itself is at fault. Credential, quota and transport failures now close the stream silently so the client reconnects and retries; a fresh websocket carries no server-side transcript, so reconnecting already implies a full context resend and needs no extra close-code signal.
Classify the failure from the upstream error body instead of the attached status. Codex reports the same cyber_policy rejection as 400 on the stream error path and as 502 through the websocket disconnect channel, so a status-only whitelist hid most of them. Treat cyber_policy as a request error so it stops credential failover without suspending the credential, and treat 413 as request-scoped because a payload that exceeds the upstream frame limit fails identically on every credential and would otherwise burn the whole pool.
Report an unroutable model as 400 invalid_request_error instead of 502 so streaming clients receive an actionable message instead of retrying forever. Keep the upstream reason in the request-log websocket timeline when the client only observes a closed connection, and stop logging expected connection-teardown races as warnings.
When a `response.completed` payload already includes output entries, fill in only missing/empty item `id`s from the streamed `output_item.done` data while keeping existing IDs untouched.
Closes: #4622
- Removed deprecated interceptor and executor-related methods, including `callRequestInterceptor`, `callResponseInterceptor`, and `callStreamChunkInterceptor`.
- Consolidated unused logic and pruned redundant imports to streamline `adapters.go`.
- No functional changes.
