Pass routing headers to ChatGPT subscription API (#62556)
## Objective Improve cache hit rate of LLM requests sent to ChatGPT subscription API. ## Solution Set headers that help route request to the correct servers. This is ChatGPT specific; plain OpenAI API doesn't require it. ## Testing I run a benchmark which imitates a thread that makes 20 sequential tool calls. I tried different cache route strategies. Results are below: | Configuration | Hit rate | |-------------------------------------------------|---------:| | No identity headers, no cache key | 22.2% | | `thread-id` only, no cache key | 44.4% | | `thread-id` + matching cache key | 42.1% | | Cache key only | 53.6% | | Turn state + cache key | 47.4% | | `session-id` + matching cache key | 100.0% | | `session-id` + `thread-id` + matching cache key | 97.4% | --- Release Notes: - Improved ChatGPT subscription caching
O
Oleksiy Syvokon committed
a8fafdd7ee36fb3fb98ebbfe5d3be983301d9e74
Parent: c05e346
Committed by GitHub <noreply@github.com>
on 8/13/2026, 1:04:16 AM