fix: group reconciliation votes by block_id to resolve same-block deadlock (#3269)
## Summary - Fixes a PoA reconciliation deadlock observed on devnet 2026-04-17 where the same block ended up on all 6 Redis nodes with three different epochs, causing permanent livelock - `unreconciled_blocks` now groups votes by `block_id` only, tracking max epoch as a tiebreaker. Identical blocks written during re-promotion storms count toward quorum. - Added a regression test that reproduces the exact production error string ## The bug During re-promotion storms (two pods racing for leadership), the same block can be written to different Redis nodes with different epochs. The old vote grouping `(epoch, block_id)` fragmented these identical blocks into separate vote groups: ``` Node state (same block_id, different epoch stamps): 1a-0, 1a-1, 1b-1: epoch 268 → vote group A, count=3 1b-0: epoch 269 → vote group B, count=1 1c-0, 1c-1: epoch 270 → vote group C, count=2 ← max-epoch winner Required quorum: 4. Winner count: 2 → repair attempted. Repair writes the winner to all 6 nodes → HEIGHT_EXISTS on every node (each has SOME entry at that height) → Written=0 → total=2 < quorum. Permanent livelock. ``` ## The fix Group by `block_id` alone; track max epoch per block_id as the tiebreaker when block_ids genuinely differ: ```rust // Before HashMap::<(u64, BlockId), (usize, SealedBlock)> vote_key = (*epoch, block.entity.id()) winner = max_by_key(epoch) // After HashMap::<BlockId, (u64, usize, SealedBlock)> // (max_epoch, count, block) vote_key = block.entity.id() winner = max_by_key(max_epoch) ``` **Behavior change:** - Same block with multiple epochs → single vote group → counts as a single block on N nodes → reconciles directly without repair (this fixes the deadlock) - Genuinely different blocks at same height → picks higher-epoch block → same behavior as before ## Test plan - [x] New test `leader_state__when_same_block_has_different_epochs_across_nodes_then_reconciles_without_repair` reproduces the exact production error without the fix (`"Backlog unresolved at height 1: repair failed to reach quorum"`) and passes with it - [x] All 9 existing `leader_state__*` tests still pass - [ ] Deploy to devnet and verify the stuck authority recovers
B
Brandon Kite committed
f7826d1c1b58cdfeabbf44bc7b671a7b14ea3039
Parent: c30e093
Committed by GitHub <noreply@github.com>
on 4/17/2026, 5:46:06 PM