SIGN IN SIGN UP

Improve cache locality reconciliation and node disk eviction (#1821)

## Summary

This is a surgical cache-locality fix across **11 existing files**. It
keeps the v1 and v2 image paths on the current architecture and does not
add or extend JuiceFS behavior.

- Include image archives and stub-code data in the node cache
disk-pressure/TTL eviction budget, while protecting active mounts and
recently used content.
- Enforce one node-local cache-server/reconciler owner per node and
quarantine stale machine-identity roots only after ownership is
established.
- Publish locality requirements only after a successful image load, then
rehydrate required v1 archives and v2 OCI layers on a failover locality
through the normal cache path.
- Align reconcile ownership grace with the 30-second cache registration
lease so a dead host cannot retain layer ownership for ten minutes after
failover.
- Treat v1 images as one archive cache object, coalesce concurrent work,
preserve the normal remote-mount path for large archives, and seed the
local cache after successful downloads.
- Classify expected v2 cache misses as misses rather than warning/error
spam; unavailable cache hosts and real failures remain visible.
- Mount the configured image-cache host path into the cache-server
DaemonSet so eviction accounts for the storage that actually fills the
node disk.

## Root cause

Cache reconciliation bounded the embedded cache but did not include
image archives sharing the same filesystem. Old per-machine cache roots
could also survive worker failover. During locality failover, a dead
cache host's registration expired after 30 seconds, but reconciliation
deferred ownership for ten minutes, leaving some HRW-selected layers
unavailable even though the new node was healthy. The legacy image path
compounded this with fine-grained cache activity instead of one
archive-level restore/seed operation.

## Scope

- 1 commit: `f7d23b1b79fc9e9613887f27ad5bf210306e767f`
- 11 modified existing files (the original draft changed 55)
- 0 added, deleted, or renamed files
- No JuiceFS, protobuf/generated, schema, or migration changes

## Validation

- `go test ./pkg/worker -count=1`
- Focused cache/image race tests
- Linux amd64 and arm64 worker compile checks
- Helm render checks for host-path and PVC cache configurations
- `git diff --check origin/main...HEAD`
- Exact multi-arch staging image build:
https://github.com/beam-cloud/beta9/actions/runs/30689780996
  - tag: `codex-locality-f7d23b1b`
- digest:
`sha256:9d0331405632eacd0a88f07fb863028539a5f22b58c388472a5043d4e01bad0b`

### Okteto staging failover proof

Ran an 8-phase cold/warm/failover probe on real workers across two
distinct staging nodes:

- worker A `20f3d9eb` on `ip-10-100-55-85.ec2.internal`
- worker B `54e1d2de` on `ip-10-100-52-235.ec2.internal`
- v1 cold/warm on A: 348 ms / effectively 0 ms
- v2 cold/warm on A: 316 ms / effectively 0 ms; 528 hits and 13 expected
first-load misses, with 0 errors/unavailable
- v1 after cross-node failover: restored from cache in 567 ms; next load
effectively 0 ms
- v2 after cross-node failover: 573 ms with 593/593 cache hits; next
load effectively 0 ms
- all 8 phases completed with 0 cache errors and 0 unavailable results
- failover worker recorded exactly one cache owner, materialized 1 v1
archive and all 6 v2 layers, and emitted 0 content-miss or
unavailable/error warning lines

The isolated Okteto setup was removed after the run and the normal
staging gateway/cache-server workloads were verified healthy.
L
luke-lombardi committed
80bcbc54d55d0db815c55be3f4ec2184046ed33e
Parent: eaae392
Committed by GitHub <noreply@github.com> on 8/1/2026, 1:35:51 PM