One of Four Nodes Stalled: Consensus After a Timeout

Four validator nodes were maintaining a single ledger. Three finalized the first block, while one remained behind. The lagging node's process was alive, and it was still connected to the other three. Yet waiting did not bring it to the same block.

This was the behavior reproduced locally in nigo-protocol on September 8, 2026. The node had neither failed to read a block from storage nor lost its network connections. In fact, the lagging node was further ahead in its attempts to reach consensus. It had timed out first, moved to the next round, and was discarding messages from earlier rounds.

Rejecting old messages sounds like a reasonable rule. But what if a message is not asking for a new vote, and instead contains evidence that other nodes have already agreed? That distinction left one live node persistently behind.

Waiting for the same block, but in different rounds

This nigo-protocol consensus configuration uses QBFT, in which a fixed set of validators exchange signed messages. With four validators, finalizing a block requires commit votes from three distinct validators for that same block. This minimum count is the quorum. One node sending the same message three times does not produce three votes.

Two numbers identify a node's position in consensus. The height is the sequence number of the block being built. The round identifies an attempt to reach consensus at that height. Moving from round 2 to round 3 while waiting for block 1 does not mean block 2 has been created. Messages and timeouts occur at different times on different nodes, so their rounds can temporarily diverge even at the same height.

Let us call the four validators A, B, C, and D. In the actual reproduction, the node represented by D was started first. We waited until its local round at height 1 reached 3, with no peers connected, before starting the others. The names are illustrative; the round and height relationships below were observed in the reproduction.

• A, B, and C: Collected commit votes for the same block in round 2 and finalized the first block.
• D: Received round 2 messages after entering local round 3. Before the fix, it discarded those messages and could not finalize the first block.

A, B, and C had three votes and could move to the next height. D kept attempting consensus in round 3 and beyond, but could not form a quorum on behalf of the three nodes that had already moved on. Its startup synchronization had also finished. Maintaining connections alone did not close the gap.

Before the fix, proposals and commit votes from a lower round were classified as stale, even at the same height. The very evidence D needed was discarded when it arrived. In the controlled reproduction, D still had block 0 when the finality deadline expired, while the other three nodes had block 1.

Going back to vote and learning a decision

The fix had a narrower goal than simply accepting old messages: allow a node to learn valid finality evidence from an earlier round at the height it is currently working on.

Among consensus messages, a PROPOSAL carries a proposed block, and a COMMIT carries a commit vote for that block. D can collect the round 2 proposal and COMMITs as finality evidence. It does not reset its own round to 2 or sign new prepare votes (PREPARE) or commit votes (COMMIT) there. Verifying signatures it has received and adding its own signature are different actions.

The checks remain in place on this path. Signatures must be valid, signers must belong to the relevant validator set, and messages must refer to the same chain and consensus context. The proposer must be eligible to propose at that height and round. A proposal after the initial round must include round-change evidence, and the rule requiring it to carry forward an earlier prepared proposal, when applicable, must also be checked.

Next, the node needs three distinct COMMITs referring to the same height, round, proposal, and block. It cannot combine one vote from round 1 with two from round 2, or mix votes for different blocks. Collecting signed hashes alone is not enough either. The node must receive the matching block body and verify its parent, height, transaction execution, and resulting state.

While evidence is incomplete, it is held in a bounded cache. The node does not finalize an unverified block or change its current voting state to follow incomplete evidence. Once the required evidence is complete, it uses the existing block commit procedure to finalize height 1 and advance to height 2. D has not retreated from round 3 to round 2. It has advanced its finalized height by one.

This distinction preserved both safety rules and progress. The current round determines which new votes the node may cast now. A complete quorum proof establishes which block has been finalized. A node's timer advancing first cannot invalidate the other validators' valid finality evidence.

Rejecting the wrong body also blocked the right one

After the initial fix, we varied message arrival order further. This exposed a separate defect: the node could reject bad input but then could not accept a corrected retry.

Suppose D receives a proposal while it is still in that round. The attached block body does not match the signed header, so validation fails. Rejecting it is correct. But the record marking the message as already processed remains.

D then times out, enters the next round, and receives three valid COMMITs. Finally, the correct body matching that same signed header arrives. Although all the evidence needed for finality is now available, duplicate detection consults the earlier record and returns DUPLICATE. A single invalid attachment has blocked valid evidence arriving later.

Turning off duplicate detection altogether would be the wrong fix. Protection against processing the same message indefinitely is still needed. The change was limited to removing the failed body attachment and its associated duplicate-tracking record. Already validated COMMITs were retained. Retransmitting just the correct body could therefore complete the evidence and allow finalization.

The same cleanup procedure now applies whether the body is rejected in the current round or arrives late after a timeout. The regression test reproduces the exact sequence: reject the invalid body, time out, receive three COMMITs, then receive the corrected body. It checks that the final input leads to finalization without creating any new signatures for the earlier round.

The rule “discard duplicate messages” does not reveal this problem on its own. Duplicate detection had to distinguish a validated proposal from the processing history of an attachment that failed validation. Alongside rejecting invalid input correctly, we needed to ask whether valid input could still be accepted afterward.

Catching up by one block is not the last check

We did not judge the fix solely by whether all four nodes reached the same height. D might have happened to enter the same round as the others and voted through the normal path. One early reproduction did exactly that, so it was excluded as evidence of learning late finality.

After the fix, we recorded both the node's local round immediately before processing a message and the round of the incoming message. In both the RDB and RocksDB experiments using the final executable, we observed D finalizing height 1 while processing a round 2 COMMIT from local round 3. The result was not obtained by allowing more time. The existing request and finality timeout settings were retained.

We then paused one of A, B, and C and submitted a new transaction only to the recovered D. With four validators and one paused, all three remaining validators are needed to form a quorum. The second block was finalized under that condition. After the paused node resumed, all four nodes converged on the same block 2 hash and state root, the hash summarizing the execution result.

This experiment showed that D did not simply obtain the first block and remain silent afterward. It did not, however, individually audit every signer in every finality certificate produced by the running processes. Separate safety tests checked the rule against signing new votes in an earlier round. In one RDB run using the initial fix, we also read the signing records after shutdown. They provided additional evidence that the node had no PREPARE or COMMIT at the first height but signed normally at the next height.

Changing the order and reopening storage

Experiments with running processes provide strong evidence that a problem was resolved in a particular execution. They cannot cover every message order at once. We separately constructed cases with proposals arriving first, votes arriving first, only the last required vote arriving late, and duplicates mixed with additional timeouts.

We varied the validator count, the proof's round, and message order, then repeated the virtual schedules to compare their transition traces. The central check was simple: incomplete evidence must produce neither finalization nor new votes; complete evidence must finalize exactly once and advance to the next height. These checks do not estimate success rates or throughput on a real network.

Separate tests closed and reopened RDB and RocksDB normally. After restoring the record that a higher round had been reached, the node still had to learn valid finality from a lower round without new signatures, then vote normally at the next height. The in-memory cache holding a partial collection of votes, however, is not restored by reopening storage. We therefore retransmitted the required proposal and votes instead of assuming that sending just the one previously missing vote would suffice.

Using the same final executable, we reran startup-skew, mTLS communication, validator synchronization recovery, and observer recovery checks on RDB and RocksDB. All selected scenarios passed.

Validation environment and scenario counts

The virtual schedules came from 144 tests combining validator counts of 4, 7, and 10, proof rounds 0 and 1, and 24 seeds for randomized ordering. Each combination ran twice, giving 288 schedules to compare. The process checks used the final executable for eight runs on macOS arm64 with Java 21, and 22 selected scenarios passed. These results were not combined with the separate four runs and 18 scenarios using the executable from before the final fix. Nor did they cover every unselected scenario in the common communication suite.

A normal close and reopen is not a forced-termination or power-loss test. The preparation conditions also differed across runs, so we did not interpret the observed durations as a measure of performance improvement. What we verified was which evidence produces which state transition.

Learning late makes participation in the next decision possible

This fix closed a gap that prevented a node from learning valid finality evidence delivered for its current height. It did not add a mechanism to locate permanently lost blocks or automatically request every vote evicted from the cache. Synchronization and recovery when the evidence never arrives remain separate problems. We also do not retrospectively attribute every previously observed timeout to this defect.

The durability rule requiring signatures to be stored before they are broadcast remains necessary. But this reproduction addressed the handling of evidence from an earlier round, rather than a wait for storage to complete. Both can look like “consensus has stalled.” Identifying the particular condition blocking progress is what makes a remedy possible.

D returned because the implementation distinguished the new votes it could cast now from the decisions it could recognize as already established. It did not need a longer wait. Carefully producing new signatures is one part of consensus. Correctly learning valid decisions that arrive late is another. Both are needed for a lagging node to participate in the next decision.