SIGN IN SIGN UP

Add off-graph KV-cache ring policy (#21447)

Summary

Adds the sliding-window (ring) layout policy. RingPolicy keeps a layer's
last window tokens in a fixed ring of window + max_write − 1 slots —
oversized so a multi-token prefill step fits without clobbering context
earlier queries in the step still need — and evicts the oldest.
SequenceCache dispatches windowed layers to it, so a mixed model
(gemma4's alternating full/sliding-window layers) shares one logical
length. Backend- and tensor-free: it emits integer write/read runs; the
per-query window mask stays in the backend.

  Files
- extension/llm/cache/cache.h — LayerPolicy::Kind::Ring; max_write on
CacheConfig (max tokens/step; default 1 = decode-only ring).
- extension/llm/cache/sequence_cache.h — split_ring_runs (wrap into ≤2
runs), RingPolicy over the oversized ring (retained_from still
window-based), the
  make_policy ring branch, and a ring-specific T ≤ max_write guard.
- extension/llm/cache/test/cache_test.cpp — ring wrap/evict, mixed
flat/ring shared length, rewind bounded by the window, and
step-over-max_write rejected.

  Testing

C++ unit tests for the ring path (wrap, eviction, mixed flat/ring,
window-bounded rewind, max_write guard). Built and ran via the standard
flow, all pass:

cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON
-DEXECUTORCH_BUILD_TESTS=ON
  cmake --build cmake-out --target extension_llm_cache_test -j
  ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure
K
Kiymet Akdemir committed
ee45eb8506941fa30f9dc2121ab8715de64048a7
Parent: 42a178f
Committed by GitHub <noreply@github.com> on 7/29/2026, 5:17:32 PM