fix: auto-disable batch API for providers that return 404 (#476)
## Summary When an upstream doesn't implement the Anthropic/OpenAI batch endpoint (e.g. MiniMax proxying the Anthropic API), every 30s flush cycle POSTs to the batch endpoint, gets a 404, and falls back to synchronous — wasting a round-trip on every cycle, forever. Onur reported this with MiniMax: `batch flush → 404 → batch fallback: processing synchronously (5/5 in the window)`. ## Changes **`packages/gateway/src/batch-queue.ts`**: - Extended `BatchProvider.submit()` return type to include `"not-found"` - Added 404 handling in Anthropic submit (line 219), OpenAI file upload (line 357), and OpenAI batch create (line 519) — all return `"not-found"` instead of falling through to the transient `null` path - Added 405 handling alongside 404 for OpenAI (vLLM, local models return 405 for unsupported endpoints) - Added `disabledBatchProviders` Set (persisted to `kv_meta` via `setKV`) — provider-level disable distinct from existing per-session disable - Provider-level bypass in `flush()`: skips batch submit entirely for disabled providers - Provider-level fast-path in `prompt()`: skips the 30s queue wait entirely for disabled providers - Cleaned up pre-existing duplicate 404/405 handlers in OpenAI submit that returned `"auth-error"` (incorrect — 404 is not an auth error) ## Design: Per-Session vs Per-Provider Disable | Signal | Scope | Rationale | |---|---|---| | 403 (OAuth scope) | Per-session | One session's token may lack batch scope while another's has it | | 404/405 (endpoint missing) | Per-provider | The upstream URL doesn't implement batches at all — affects everyone | Both are persisted to `kv_meta` and survive process restarts.
B
Burak Yigit Kaya committed
0684cd783cdb50670b89ce06d8775caf89627364
Parent: ac3d3cf
Committed by GitHub <noreply@github.com>
on 5/27/2026, 2:15:25 PM