When HTTP responds but consensus stalls
A transaction has been submitted, but it has not finalized. The operations dashboard opens, and status queries succeed.
The consensus engine reports running=true. If the server responds and the engine is running,
is waiting a little longer all that is needed?
That information cannot answer the question yet. The node might be idle because there is no work to do, waiting for votes from other validators, or stuck inside a storage operation. The value still on the screen might even be the last response received before a failure.
The article on GC safety and progress distinguished stopping to protect data from finishing the cleanup. Observing consensus raises a similar question: How do we distinguish the absence of a visible failure from evidence that the work is getting done?
nigo-protocol does not leave that question to response status alone. It separates the path for reading minimal execution status, then assesses progress through distinct observations of pending transactions, votes, and local finalization. This article covers that design and the scope established by local validation in September 2026.
The same height can mean different things
For illustration, suppose four validators have finalized the same blocks through height 10. A height is a block's position in the sequence. This network does not produce empty blocks when there are no transactions to process. Right now, it has no pending transactions or block proposals being processed.
If the height is still 10 a minute later, is that a failure? Under these conditions, it is normal idle time. There was nothing to finalize, so no new block was needed. An alert based only on “a long time has passed since the last block” would label normal operation a failure whenever transactions are infrequent.
Now submit transaction T. It enters one node's transaction pool, and that node accepts proposal P, which contains T in a block at height 11. There is now something to wait for. In this example, finalization requires three votes, but suppose the node has observed only two valid commit votes for that proposal. Having three network connections does not mean it has received the missing third vote.
If this condition lasts beyond the observation threshold for a delay, there is evidence to report that the node is waiting for votes. But its vote count alone cannot establish whether the cause is another node's failure, a communication delay, or a processing result that has yet to arrive.
Once the required votes arrive, the node completes local ledger finalization and stores the consensus safety record. It can then report finalization at height 11. If T leaves the pool and there is no next proposal, the node becomes idle again. Both idle periods are quiet, but between them there is evidence that the submitted work actually finished.
These heights and transaction names are illustrative. The actual experiment results appear separately below.
HTTP success and a running engine answer different questions
The information available to the operator in this example has three layers.
| Observation | What it establishes | What it does not establish |
|---|---|---|
| Successful HTTP response | This query received a response | Whether consensus is progressing |
| Engine running | Local execution conditions hold | Whether enough votes have arrived |
| New local finalization | Finalization and storage finished at that height | Whether later transactions will keep finishing |
A changing response timestamp means a new query was made.
It is not a signal that consensus produced a new result. Likewise, running describes an execution state:
the engine has started and has not been classified as explicitly failed or stopped.
That value alone cannot rule out an engine that waits a long time without throwing an exception.
Conversely, the absence of a new block does not establish that the engine has stopped. Normal idle time exists, as in the height-10 example with no work to do. Alongside “What has changed recently?”, we must ask whether there is work that should be finishing now.
A new finalization observation is still a result confirmed by one node. It does not establish the current state of every node in the network or guarantee long-term availability. An operations dashboard should present observed facts, not declare unobserved conditions healthy on their behalf.
Keeping the observation path out of the same blockage
Before distinguishing progress states, there was another problem to solve. If a status request also reads detailed ledger data, it can end up waiting on the same storage that has slowed down. The same applies to diagnostics that must acquire a lock inside the consensus engine. The path intended to explain a failure then depends on the troubled operation finishing first.
nigo-protocol separates minimal status into its own read path. The engine publishes its execution state and first failure in a small in-memory snapshot, and queries read the already published snapshot. They do not reacquire the consensus processing lock or access the database, and they do not query the transaction pool or peer connections.
Reporting a failure also avoids reacquiring the failed state machine's lock. If a transition fails and its resulting state cannot be confirmed, those fields remain empty. Showing the confirmed failure first is more useful than delaying its publication to fill every field.
Detailed queries still exist. Screens that need ledger contents still read storage. The difference is that minimal execution status remains separately available while those requests fail or wait.
Isolated validation on September 14 exercised this separation with real HTTP requests. Minimal status queries responded while detailed requests waited for an exhausted JDBC connection pool or reads were blocked by a RocksDB lock. Separate checks covered detailed-query failures after storage was closed. These results show that the observation path is independent of those storage dependencies. They do not guarantee a response if the JVM itself stops or CPU or HTTP processing resources are exhausted.
Count the work waiting and the work actually finished
Separating minimal status cannot, by itself, explain the example waiting for votes. Assessing progress requires three further kinds of evidence.
The first is transactions still in the pool and how long they have waited. Admission to the pool does not mean a transaction is immediately executable. Rather than declaring a consensus failure just because many transactions are pending, the system separately reveals whether any have waited a long time. Other transactions might still be finalizing.
Waiting time is measured from this node's first successful admission of the transaction, not from the signing timestamp inside it. Receiving the same transaction again does not move the start of its wait forward. Otherwise, T could appear new every time it was resubmitted, despite having waited for a long time.
The second is valid votes for the same height, round, and proposal. A round is another attempt to reach consensus at the same height. Repeated copies of one validator's vote count only once, and votes for different proposals are not combined to give the appearance of a quorum. The waiting time for consensus work at height 11 does not restart simply because the round changes.
The third is the height finalized locally and the time elapsed since that completion. Work remains between collecting enough votes and finishing storage. Finalization is observed as complete only after the ledger is finalized and the required consensus safety record has been stored. Timer executions and repeated processing of the same message do not count as finalizations.
The current progress assessment reads this evidence together. An explicit failure or an overdue local operation or timer takes precedence. Otherwise, when consensus work has remained unfinished too long, the assessment distinguishes waiting for votes from delayed finalization. It also separately identifies transactions that have remained pending too long. If there are neither proposals nor pending transactions, the state can be read as normal idle time.
The current default threshold for delayed progress is 60 seconds. It classifies observations; it does not change the consensus quorum or transaction validity, or require forced finalization after 60 seconds. Detecting a delay on the operations dashboard does not automatically restart the node or bypass stored safety records.
Do not fill gaps in knowledge with a healthy status
Each source publishes an internally consistent snapshot, but together they are not a photograph taken by freezing
the entire transaction pool and consensus state at one instant. Reading while consensus moves to another round
can briefly combine information from incompatible moments. In that case, the result is UNKNOWN—the available
evidence cannot establish the current status—rather than a plausible-looking healthy assessment.
Immediately after a process restart, the system also does not invent an elapsed time since the last finalization in the previous run. If the new process has not observed a finalization, the value remains absent. “Not yet observed” and “finalization has been delayed for a long time” are different conditions.
The same principle applies on screen. Once the last received healthy value is more than 15 seconds old, it is no longer evidence of current health. Query failures, paused updates, and queries being refreshed also require the previous sample to be distinguished from the current assessment. Those 15 seconds govern the freshness of the displayed sample, a different purpose from the 60-second progress threshold above.
Here, UNKNOWN does not establish a failure either.
It prevents a disconnected screen from continuing to announce healthy consensus.
Being able to express a lack of knowledge is necessary to distinguish an actual failure from a gap in observation.
What submitting transactions to four quiet nodes established
On September 16, four validator processes were started from the same node JAR to check whether idle time without transactions and the finalization of new transactions were distinguished correctly. The run used an RDB backend and mutual TLS connections in a local macOS environment, with empty-block production disabled.
During roughly 65 seconds of idle time, progress was sampled 63 times across the four nodes.
All 252 samples reported IDLE. There were no pending transactions, and because no finalization had yet
been observed, the elapsed time since finalization was absent too.
Consensus rounds changed, but those changes were not mistaken for new finalizations or delayed progress.
Two transactions were then submitted through different nodes. All four nodes finalized the same chain through height 2, and each node's last locally observed finalized height was also 2. After processing the transactions, they returned to having no pending transactions. The run distinguished quietness before any work arrived from quietness after the work had finished.
This experiment checked normal idle time and new finalizations. The classification of insufficient votes and storage delays was checked separately through deterministic Java tests and isolated HTTP validation. The 252 samples are not evidence from a production environment undergoing multiple failures, nor do they establish long-term network availability or automatic recovery. The experiments were not rerun while writing this article.
The purpose of operational observation is not to keep a green indicator on for as long as possible. It is to distinguish what is waiting, what has actually finished, and what is currently unknown. That distinction helps avoid unnecessary intervention in normal idle time while exposing delays behind a responding server.
Can We Send the Signature Once the Save Call Returns? examines the storage boundary touched on here. What must be stored before a consensus message can be sent to another node, and does a storage function returning establish that the requirement has been met?