- Add `upsertAuthResult` to incrementally update only affected model shards after request completion.
- Cache supported model sets per auth using registry epoch tracking to avoid redundant lookups.
- Preserve full shard synchronization for auth lifecycle events and credential-scoped results.
Closes: #5061
- Reuse coresession.ExtractSessionInfo across HTTP headers and request payloads to unify canonical session prefix namespaces with the scheduler.
- Extract hierarchical session identities in two phases: initial extraction from request headers on entry, and authoritative deep extraction once request payloads and metadata are available.
- Support Claude Code multi-level subagents (X-Claude-Code-Agent-Id, metadata.agent_id) and Codex thread fork lineages.
- Propagate SessionID and ParentSessionID across ClientRequestMetadata, UsageReporter, and coreusage.Record without root_session_id.
- Include session_id and parent_session_id in queuedUsageDetail for Home LPushUsage forwarding and Redis consumption with self-loop guards.
- Add comprehensive test coverage for canonical headers, body extraction, ghost parent elimination, and self-referential loop guards.
- Mark Codex usage limit errors as credential-scoped across HTTP and WebSocket executors.
- Support both top-level and nested error structures with case-insensitive matching when parsing retry-after resets.
- Propagate prevalidated candidate context to session affinity and built-in selectors during auth selection.
Closes: #5529
- Implement three-way merge for refreshed and prepared auth updates against base and current runtime state.
- Retain user modifications to metadata, attributes, proxy URL, and error/cooldown status across background refresh operations.
- Guard against stale registration epochs and enforce per-auth generation ordering during persistence.
Closes: #5465
- Add routing.session-affinity-subagents (defaulting to true) to RoutingConfig.
- Unify subagent parent credential inheritance across all providers by removing hardcoded provider/model blacklists.
- When session-affinity-subagents is false, isolate subagents and distribute them via the fallback selector.
- Ensure changing session-affinity-subagents is a no-op when session-affinity is false.
- Preserve alias isolation and failure isolation invariants.
Closes: #5417
- Enforce a 10-second minimum cooldown floor for quota errors to prevent retry storms from sub-second `Retry-After` values.
- Track attempted authentication credentials per round across execution flows.
- Ensure credentials attempted during a failed 429 round respect the cooldown floor instead of triggering zero-wait retries.
Closes: #5265
- Map `user_credits_insufficient` to `402 Payment Required` in `decodeHomeDispatchError`.
- Map `user_period_limit_exceeded` to `429 Too Many Requests` in `decodeHomeDispatchError`.
- Add unit test coverage for billing and period limit error mappings.
Closes: #5170
- Parse and prioritize JWT `exp` claims when determining access token expiration.
- Retain active credentials and schedule backoff retries on refresh failures if the access token remains unexpired.
- Demote and block auth credentials with expired access tokens during model scheduling and candidate selection.
- Reduce Codex token refresh lead duration to 24 hours.
Closes: #5095
- Bypass short cooldown checks and recording in execution flows when cooling is disabled globally or per auth.
- Skip marking credits permanently disabled and refreshing credit hints when cooling is disabled.
- Export quota cooldown status helper functions for auth and configuration evaluations.
Closes: #4793
- replace O(N log N) sorting under global write lock with container/list LRU eviction
- track primary alias groups for O(1) group lookup and clean group eviction
- capture timestamps inside lock and add nil receiver guards
- Extract subagent IDs, user IDs, and prompt cache keys from nested `request` objects across session identity, session info, and auth selection.
- Normalize and fall back appropriately when top-level prompt cache key fields are empty strings.
- Inspect nested request fields when validating explicit session identities.
- Introduce `HTTPStatusError` to retain upstream HTTP status codes across Antigravity OAuth and project lookup calls.
- Propagate status codes and retry-after metadata from cause errors in `missingAntigravityProjectIDError`.
Closes: #5368
- Attach candidate upstream errors as causes to scheduler availability and cooldown errors.
- Introduce `errorWithCause` wrapper to extract and display upstream error summaries.
- Enrich auth selection error messages with upstream details and preserve model cooldown errors.
- Forward `Retry-After` response headers for model cooldown errors in Claude Code handler.
Closes: #5365
- Prevent subagent sessions from inheriting parent credentials on Antigravity and Gemini providers and models to mitigate concurrency rate limiting.
- Avoid aliasing subagent session keys to parent fallback keys or mutating parent bindings during subagent result handling.
Closes: #5364
- Implement Merkle Longest Common Prefix (LCP) session affinity engine across OpenAI, Claude, Gemini, Interactions, and Responses protocols in sdk/cliproxy/session/lcp.go
- Implement in-memory hierarchical session tree store with dynamic lineage recalculation, reparenting, cycle prevention, and parentIndex descendant cascading in sdk/cliproxy/session/tree.go
- Integrate Merkle prefix matching and hierarchical session tree into SessionAffinitySelector
- Bound SessionCache capacity to 65,536 entries with deterministic eviction
- Support Claude nested metadata parent extraction and Header+Body parent merging
- Enforce TTL expiration checks in touchLocked and bindLocked
- Propagate ParentSessionID in authDispatchRequest and RPopAuthWithSessionHierarchy for distributed Home dispatch
- Add isHierarchyParent in home_session_alias.go covering Claude, Codex, Pi slot, Antigravity, and Gemini caching
- Add comprehensive test suites in lcp_test.go, tree_test.go, selector_lcp_test.go, client_test.go, and home_session_alias_test.go
- Add `RegistrationEpoch` and `Generation` tracking to `Auth` and model registry to prevent stale scheduling and snapshot reconciliation races.
- Implement scheduler removal tombstones and `ApplyClientModelProjections` in the model registry to atomically discard out-of-order state updates.
- Track logical route models across execution results and preserve active model cooldowns during registry reconciliation.
Track when executor calls cross an upstream transport boundary and use that
signal to keep model/provider errors from being replaced by later local
preparation, selection, or internal failures.
Mark HTTP, websocket, relay, and usage-tracked transports as upstream
attempts, while avoiding marks for local validation, logging, missing
sessions, and successful websocket handshakes before request send.
Parse relative auth expiry metadata and adjust Antigravity refresh timing.
Keep round-robin rotation stable in the scheduler fast path when
credentials enter cooldown or are removed. Replace the numeric readyView
cursor modulo normalization with ID-based successor binary search,
aligning readyView with RoundRobinSelector.
Both built-in rotation strategies lost their position whenever the candidate
set shrank, which happens on every retry that excludes an already tried
credential and on every cooldown transition.
Smooth weighted round-robin reset every accumulated credit as soon as the
weight vector differed from the previous call. A transient subset is not a
configuration change, so the reset fired constantly. With all credits back at
zero and equal weights, the strict `>` comparison always resolves ties to the
first entry in slice order, and candidates are sorted by auth ID, so every
retry restarted the cascade at the alphabetically first credential. Credits are
now reset only when a credential's configured weight actually changes, and the
accumulator is bounded so permanently removed credentials cannot leak entries.
Plain round-robin indexed a monotonic counter into the filtered slice via
`available[index%len(available)]`. Once the slice shrank, the modulo re-seated
the rotation instead of resuming it. Rotation now continues from the identity
of the previous pick, resolved with a binary search over the sorted ring.
Measured over 9 equally weighted credentials with a realistic mix of long
sessions, new sessions and retries, the busiest-to-quietest ratio drops from
11331x to 1.1x. Distribution is exact for equal weights, matches configured
ratios for unequal weights, and tracks the theoretical optimum within 3% under
random credential unavailability.
Codex and Claude already emit credential-level quota watermarks on ordinary
responses. CPA used to drop them. Keep the latest watermark in memory and
return it from the management auth-file API.
Hard rule: this is observation only. It must not change scheduling, cooldown
selection, or auth-file persistence.
Snapshot, not accumulation
- QuotaState now has ObservedAt and a bounded Signals map. MarkResult fills
them from the response headers already recorded on the request.
- Signals is the current response, not a union of earlier ones. Retry-After
and "limit reached" only appear on the response that produced them; merging
across responses would keep an expired value forever.
- A response with no quota header (transport failure, 5xx, unrelated endpoint)
leaves the previous snapshot in place.
- ObservedAt is the time of the current snapshot. It advances even when the
values did not change, so a consumer can tell a fresh reading from a stale
one.
- When two model states merge, keep the newer snapshot. Do not union keys
captured at different times.
What is observed, and what is not
- One predicate, ProviderSupportsQuotaObservation, decides the provider set.
- Keep Codex and Claude. Drop Kimi, xAI/Grok, Antigravity, and the Gemini
family (gemini/vertex/aistudio): their ordinary headers are not a reliable
credential-level remaining quota.
- Count-tokens reuses the credential but is not generation traffic.
ExecuteCount sets SkipQuotaObservation so those headers cannot replace the
last generation snapshot. Cooldown and success/failure accounting still run.
Cooldown must not overwrite the last snapshot
- Observation writes only ObservedAt and Signals.
- Cooldown writes only Exceeded, Reason, NextRecoverAt, and BackoffLevel,
through applyCooldownFields. Never assign a fresh QuotaState{...} over a
live value: that would zero the snapshot on 429, Cloudflare, credential-
scope sibling updates, and cooldown clears.
- If a credential-quota cooldown is still active, MarkResult still observes
an already-present model state. It does not create scheduler state just to
record a watermark.
- .cds files persist cooldownFieldsOf(Quota) only. Restore keeps the newer
ObservedAt, so reloading cooldown cannot clobber a newer in-memory snapshot.
- cooldownQuotaEqual still ignores observation fields, so a watermark change
cannot by itself persist cooldown or move the scheduler.
- The management payload omits every cooldown field, so it cannot be mistaken
for scheduler state or wired back into scheduling.
- Manual ResetQuota still clears the full QuotaState.
Codex websocket events
- Codex WS reports quota as codex.rate_limits frames, not HTTP headers.
ParseCodexQuotaEventHeaders turns one event into the same bounded header
shape, and MergeResponseHeaders folds it into the request-scoped holder.
additional_rate_limits is accepted as an object (websocket) or an array
(/wham/usage).
- Parse only through AppendCodexAPIWebsocketResponse. The shared
AppendAPIWebsocketResponse is also used by xAI, and xAI error frames really
do carry x-ratelimit-* headers. Parsing every frame as Codex quota would
forge Codex headers into another provider's request log.
- Also capture code_review_rate_limits.
- A malformed active-limit name drops only that one header, not the window
watermarks parsed from the same event.
- The type discriminator scans a bounded frame prefix, not every byte of
every frame.
- HTTP namespaces an extra limit by short name (x-codex-bengalfox-*); WS
namespaces it by limit name (GPT-5.3-Codex-Spark). The two paths cannot
emit the same header names. The X-Codex-Additional- prefix marks the WS
origin, and snapshot replacement keeps the two spellings from piling up.
Hardening
- Reject observed values with control characters. These strings reach the
plain-text request log, and Limit-Name is upstream-controlled, so CR/LF
could forge a header line.
- When the header cap is hit, keep plan/credits/primary ahead of
additional-limit namespaces, then sort names so truncation is deterministic.
- QuotaState.Clone deep-copies Signals and is used by Auth.Clone and
ModelState.Clone.
- Token stores still serialize credential metadata only, so observation adds
no auth-file writes.
Redefine request-retry as additional credential retry rounds and
enforce max-retry-credentials per round. Home dispatch now carries
excluded and pinned auth constraints, supports remote retry limits, and
propagates cooldown retry-after metadata across exhausted rounds.
Move Antigravity upstream retries under conductor ownership to avoid
double-consuming retry attempts. Update configuration comments and add
coverage for Home retry rounds, cooldown handling, pinned credentials,
and legacy dispatcher compatibility.
* feat(codex): add opt-in stream bootstrap buffering
The upstream smuggles capacity rejections into an HTTP 200 stream. The
handshake events arrive normally and only a later event carries
{"error":{"type":"service_unavailable_error","code":
"server_is_overloaded"}}. By then the executor has already handed the
first chunk downstream, the response is committed, and the conductor can
no longer retry on another credential, so the request fails even though
other credentials were available.
When codex.stream-bootstrap-buffering is enabled the executor holds back
the handshake events until it can tell whether the stream carries real
output or a rejection. An overload rejection then fails the attempt
before any chunk is delivered, letting the conductor retry on another
credential; every other terminal failure is flushed in order and
delivered in-stream exactly as before.
Detection uses an event-type allow-list rather than a fixed count. On the
websocket transport codex.rate_limits and codex.response.metadata arrive
before response.created, making the first generated event the fifth
frame, so a small counter would release the stream before the rejection
is visible. Buffering is bounded and hitting the bound degrades to the
original unbuffered behaviour.
Two details are load-bearing. The error must be returned synchronously:
delivering it as the first stream chunk makes ExecuteStream downgrade it
into a committed 200 and the status is lost. And the websocket path must
not signal an upstream disconnect for a rejection it intends to retry,
because the downstream handler closes the client connection on that
signal and the retry would have nowhere to deliver.
The 503 status is produced only on this path rather than in the shared
codexTerminalFailureStatus mapping, so disabling the feature restores the
previous behaviour exactly, including cooldown classification and
retry-after parsing.
Defaults to false: response headers are withheld until generation
starts, which can trip client or reverse-proxy read timeouts.
* test(codex): pin bootstrap overload failover through the conductor
Executor-level tests cannot show what the client finally receives. These
exercise ExecuteStream end to end to pin three properties that are easy
to regress:
- consecutive overloaded credentials are skipped until one serves the
request, and retries are capped by max-retry-credentials rather than
multiplying with request-retry
- exhausting the pool surfaces the upstream status instead of a
committed 200 stream
- with buffering disabled the rejection stays an in-stream error on a
committed stream, which is the behaviour the feature must preserve
The third case also documents why the executor returns its error
synchronously: an error arriving as the first stream chunk is wrapped and
downgraded into a committed 200, silently losing the status.
- Preserve API key entries with empty `api_key` when `base_url` is configured, and extend config dedupe/ID logic to include base URL, proxy, prefix, and headers so identities are stable.
- Update config/auth handling so `auth_kind=apikey` is treated as config API-key auth even without an `api_key` field, enabling base_url-only credential records.
- Ensure Gemini/Codex/XAI request path clears `Authorization`/provider auth headers when token is empty to avoid leaking unrelated auth state.
- Add `oauth-request-scoped-errors` configuration with normalization, sanitization, and YAML management persistence/hot-reload hooks.
- Route request-scoped error classification to use per-provider rules only for OAuth auth entries.
- Add config diff reporting and management CRUD endpoints for `oauth-request-scoped-errors` (get/put/patch/delete) with input sanitization.
Closes: #5085
- Add compact-specific error classification to mark transient/non-credential failures as availability-neutral instead of triggering cooldown penalties.
- Stop auth fallback immediately on compact request-fault errors (e.g., bad/not-found/unsupported request errors) and return the upstream compact error.
- Preserve existing cooldown behavior for auth/credential faults (`401`, `403`, `429`) while allowing non-auth compact failures to fail fast without tainting normal traffic routing.
Closes: #5031
- Changed `DisableCooling` from a boolean to a pointer in various config types to allow explicit inheritance.
- Updated tests to reflect the new pointer usage for `DisableCooling`.
- Enhanced the `BuildConfigChangeDetails` function to handle optional boolean changes for `DisableCooling`.
- Added new tests to ensure proper handling of cooling overrides in configurations.
- Refactored the `SetQuotaCooldownDisabled` function and related logic to clarify the purpose of cooldown management.
- Introduced new tests for cooling override precedence in the auth manager.
- Ensured that all relevant handlers and synthesizers correctly manage the `DisableCooling` setting.
- Add `isSameSelector` using type-aware comparable checks to avoid unnecessary selector replacement.
- Update `Manager.SetSelector` to:
- serialize swaps with a dedicated selector mutex,
- no-op when replacing with the same selector instance/type,
- stop the previous selector when it implements `StoppableSelector`.
- Protect `SessionCache.Stop()` with `sync.Once` and nil-check to make repeated/concurrent stops safe and idempotent.
Closes: #5018
- Canonicalize model IDs when building session-affinity cache/fallback keys so variant suffixes (for example thinking modes) map to the same binding.
- Normalize model values from result metadata/on-result release paths to release and rebind bindings consistently across canonical model keys.
Closes: #5016
- Add request-scoped error rule extraction from auth metadata or runtime provider config (including OpenAI compatibility fallback)
- Match rules by HTTP/status-code plus error body substring or regex patterns
- Support `stop`, `stop-and-cooldown`, `continue`, `continue-and-cooldown` actions with normalized validation
- Apply matched actions to execution results via request-scoped vs force-cooldown error codes and stop/continue flow control
- Introduce request-stop error wrappers/helpers for matching and unwrapping scoped stop state
Closes: #5006
- Propagate session-affinity provider/model metadata through selection and execution results so affinity cleanup uses the same namespace/model keys as selection.
- Forward execution `Options` in `Result` across normal, home, and stream paths, then hook manager cooldown handling to notify the affinity selector.
- Add `Touch`/`CompareAndDelete` to session cache and switch affinity release logic to compare-and-delete semantics, preventing stale failures from deleting a session already rebound to another auth.
Closes: #4989
- Treat request-invalid/invalid-parameter 401 faults as request-scoped so auth/model cooldown is skipped.
- Export request-scoped/connection-lifecycle error codes and add helpers to explicitly create or mark request-scoped `Error`s.
- Improve request-fault detection by checking the raw message body when `Error.Code` formatting hides JSON payloads.
Closes: #4976
- Added metadata merge helpers in `sdk/cliproxy/auth` to retain existing auth-file settings while excluding token lifecycle fields.
- Applied merge before saving in management handler and `sdk/auth.Manager.Login` so re-login overwrites credentials without discarding user-configured fields.
- Updated token save paths for Claude, Codex, Kimi, XAI, and Vertex to persist merged metadata payloads when writing auth files.
Closes: #4945
- Added a plugin refresh-compat executor wrapper that forwards normal OpenAI-compat execution paths while routing `Refresh` to plugin `AuthProvider`/Home refresh logic.
- Updated refresh lookup to use the effective executor key from auth metadata so namespaced compatibility providers can resolve their refresh executors correctly.
- Changed OpenAI-compat registration to wrap built-in executors with the plugin-refresh wrapper when a matching plugin auth provider exists, while preserving bare executors otherwise.
- Made `OpenAICompatExecutor.Refresh` fail fast for OAuth-style credentials (with refresh tokens) instead of silently returning unchanged auth.
Closes: #4719
