Improve cache locality reconciliation and node disk eviction (#1821)
## Summary This is a surgical cache-locality fix across **11 existing files**. It keeps the v1 and v2 image paths on the current architecture and does not add or extend JuiceFS behavior. - Include image archives and stub-code data in the node cache disk-pressure/TTL eviction budget, while protecting active mounts and recently used content. - Enforce one node-local cache-server/reconciler owner per node and quarantine stale machine-identity roots only after ownership is established. - Publish locality requirements only after a successful image load, then rehydrate required v1 archives and v2 OCI layers on a failover locality through the normal cache path. - Align reconcile ownership grace with the 30-second cache registration lease so a dead host cannot retain layer ownership for ten minutes after failover. - Treat v1 images as one archive cache object, coalesce concurrent work, preserve the normal remote-mount path for large archives, and seed the local cache after successful downloads. - Classify expected v2 cache misses as misses rather than warning/error spam; unavailable cache hosts and real failures remain visible. - Mount the configured image-cache host path into the cache-server DaemonSet so eviction accounts for the storage that actually fills the node disk. ## Root cause Cache reconciliation bounded the embedded cache but did not include image archives sharing the same filesystem. Old per-machine cache roots could also survive worker failover. During locality failover, a dead cache host's registration expired after 30 seconds, but reconciliation deferred ownership for ten minutes, leaving some HRW-selected layers unavailable even though the new node was healthy. The legacy image path compounded this with fine-grained cache activity instead of one archive-level restore/seed operation. ## Scope - 1 commit: `f7d23b1b79fc9e9613887f27ad5bf210306e767f` - 11 modified existing files (the original draft changed 55) - 0 added, deleted, or renamed files - No JuiceFS, protobuf/generated, schema, or migration changes ## Validation - `go test ./pkg/worker -count=1` - Focused cache/image race tests - Linux amd64 and arm64 worker compile checks - Helm render checks for host-path and PVC cache configurations - `git diff --check origin/main...HEAD` - Exact multi-arch staging image build: https://github.com/beam-cloud/beta9/actions/runs/30689780996 - tag: `codex-locality-f7d23b1b` - digest: `sha256:9d0331405632eacd0a88f07fb863028539a5f22b58c388472a5043d4e01bad0b` ### Okteto staging failover proof Ran an 8-phase cold/warm/failover probe on real workers across two distinct staging nodes: - worker A `20f3d9eb` on `ip-10-100-55-85.ec2.internal` - worker B `54e1d2de` on `ip-10-100-52-235.ec2.internal` - v1 cold/warm on A: 348 ms / effectively 0 ms - v2 cold/warm on A: 316 ms / effectively 0 ms; 528 hits and 13 expected first-load misses, with 0 errors/unavailable - v1 after cross-node failover: restored from cache in 567 ms; next load effectively 0 ms - v2 after cross-node failover: 573 ms with 593/593 cache hits; next load effectively 0 ms - all 8 phases completed with 0 cache errors and 0 unavailable results - failover worker recorded exactly one cache owner, materialized 1 v1 archive and all 6 v2 layers, and emitted 0 content-miss or unavailable/error warning lines The isolated Okteto setup was removed after the run and the normal staging gateway/cache-server workloads were verified healthy.
L
luke-lombardi committed
80bcbc54d55d0db815c55be3f4ec2184046ed33e
Parent: eaae392
Committed by GitHub <noreply@github.com>
on 8/1/2026, 1:35:51 PM