Add off-graph KV-cache ring policy (#21447)
Summary Adds the sliding-window (ring) layout policy. RingPolicy keeps a layer's last window tokens in a fixed ring of window + max_write − 1 slots — oversized so a multi-token prefill step fits without clobbering context earlier queries in the step still need — and evicts the oldest. SequenceCache dispatches windowed layers to it, so a mixed model (gemma4's alternating full/sliding-window layers) shares one logical length. Backend- and tensor-free: it emits integer write/read runs; the per-query window mask stays in the backend. Files - extension/llm/cache/cache.h — LayerPolicy::Kind::Ring; max_write on CacheConfig (max tokens/step; default 1 = decode-only ring). - extension/llm/cache/sequence_cache.h — split_ring_runs (wrap into ≤2 runs), RingPolicy over the oversized ring (retained_from still window-based), the make_policy ring branch, and a ring-specific T ≤ max_write guard. - extension/llm/cache/test/cache_test.cpp — ring wrap/evict, mixed flat/ring shared length, rewind bounded by the window, and step-over-max_write rejected. Testing C++ unit tests for the ring path (wrap, eviction, mixed flat/ring, window-bounded rewind, max_write guard). Built and ran via the standard flow, all pass: cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON -DEXECUTORCH_BUILD_TESTS=ON cmake --build cmake-out --target extension_llm_cache_test -j ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure
K
Kiymet Akdemir committed
ee45eb8506941fa30f9dc2121ab8715de64048a7
Parent: 42a178f
Committed by GitHub <noreply@github.com>
on 7/29/2026, 5:17:32 PM