Skip to content

fix: make full nodes not resync infinitely on flush - #614

Open
pablocampogo wants to merge 2 commits into
developmentfrom
fix-resync-on-flush
Open

pablocampogo wants to merge 2 commits into
developmentfrom
fix-resync-on-flush

Conversation

@pablocampogo

Copy link
Copy Markdown
Collaborator

Fix full-node stall: trigger an active resync when the block inbox backs up

Summary

Full nodes (non-validators) could permanently stall on new block messages and only recover after a restart. The existing 80% inbox dead-letter flush did not fix it — in one production incident the flush fired 16 times over ~2 hours and the node still never resynced. This PR makes a backed-up block inbox trigger an active resync instead of just dropping the backlog.

Root cause

Inbound BLOCK messages funnel into a single shared channel consumed by either ListenForBlock() (steady state) or Sync() (catch-up). The only pre-existing way for a synced full node to re-enter Sync() was the syncDetector (NewHeightTracker) in ListenForBlock(), which requires ≥ 1/3 of peers to each deliver a fresh ErrNewHeight block.

That recovery path structurally starves exactly when it's needed:

  • When a full node falls behind, the shared block inbox saturates. The non-blocking producer drops the newest messages, and the DLQ drops the entire backlog at 80%.
  • Those drops discard the very ErrNewHeight gossip that syncDetector needs to count toward its 1/3 threshold, so the trigger never fires.
  • Gossiped blocks are de-duplicated (MessageCache) and applied strictly in-order, so any height dropped by the flush never arrives again on its own. Only Sync() re-requests specific heights — and there was no path from "I dropped blocks" → "request them back."

Net effect: the node churns (flush → refill → flush) forever and never transitions to sync. A restart worked only because boot unconditionally runs Sync(). This was full-node-only because the DLQ is disabled for validators.

Additionally, the flush was actively harmful during a sync: it would drop the in-order block responses Sync() depends on, causing timeout/re-request livelock.

Changes

  • p2p/p2p.go: added a syncing flag (SetSyncing/IsSyncing) and a coalesced mustResync channel exposed via MustResync() / signalResync().
  • p2p/conn.go (dropInboxIfBackedUp): for the BLOCK topic on a full node, do nothing while a sync is already running (so Sync() keeps its in-order responses, matching validator behavior), otherwise signal a resync before draining the backlog.
  • controller/block.go (ListenForBlock): now selects on both Inbox(Block) and MustResync(); on a resync signal it launches Sync() and cleanly hands off consumption.
  • controller/consensus.go: Sync() is idempotent via isSyncing.CompareAndSwap (prevents duplicate Inbox(Block) consumers now that there are multiple trigger paths) and toggles P2P.SetSyncing(true/false) in Sync()/finishSyncing().

Behavior after fix

  • A full node that falls behind now switches to an active resync that re-requests missing heights in order, rather than relying on the peer-count heuristic that the backup itself was starving.
  • The DLQ flush is suppressed during sync, so full-node sync behaves like validator sync (no dropped responses).
  • Validators remain fully exempt from the DLQ.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant