How Can a Stale “Healthy” Status Hide a Failure?
At 10:00 a.m., a node's operations screen shows ordinary readings. Memory usage is close to its usual level, and no work is waiting. You turn to something else and return at 10:03. The screen looks the same. Has the node continued to run smoothly?
The answer changes if status requests have failed in the meantime. The values on the screen are records from 10:00. You cannot tell whether the node is still in the same state, or whether the node is fine and only the observation path has failed. Continuing to display an earlier healthy sample is different from declaring the node healthy now.
The article on HTTP responses and actual consensus progress examined how transaction finalization can stall even while a server responds. This article turns to the screen that presents that evidence. Even correct observations can lead to wrong conclusions when their time and scope are lost. Preserving those boundaries on screen was a starting point for nigo-protocol's operations console implementation in September.
The last value received needs a timestamp
Consider the situation more closely. This timeline illustrates the behavior; it is not an incident record. All times in the table use a single clock.
| Time | What happened | What the screen can establish |
|---|---|---|
| 10:00 | A status request succeeded | The sample received then showed no signs of trouble |
| 10:01 | Another request received no response | The new state could not be checked |
| 10:03 | The operator reads the same screen | The last sample is from 10:00; the current state is unconfirmed |
Clearing every old value would also remove clues about the moments before the failure. Leaving those values without explanation, however, makes them look current at 10:03. For ordinary refresh failures, the console therefore retains the previous values while displaying the request error and a notice that this is the last successful response and may differ from the current state. A failed request does not move the last successful response time forward.
There is more than one relevant timestamp. The browser's “Last success” is the time it received a successful response. The server observation time in a process or storage response identifies when the server collected the values. A slow request can create a gap between the two, and the computers' clocks may differ. Simply subtracting these timestamps does not give a reliable measure of network delay.
The screen therefore keeps “a response just arrived” distinct from “the current state was just measured.” Alongside a value, an operator needs to know when it was collected and when it arrived.
Filling an empty field with zero changes the conclusion
Suppose the screen cannot obtain CPU utilization at 10:03. Putting 0% in the field makes the display look tidier. But it changes the meaning from “the load could not be measured” to “the load was measured and was zero.” Similarly, showing 0 records when a storage count could not be collected turns a failed observation into a claim that the store is empty.
The nigo-protocol console leaves CPU readings unmeasured before an initial measurement or when the metric is unsupported. It does the same for storage counts it could not collect. “Unmeasured” is not a failure diagnosis either. It identifies what was not observed. A failed storage metrics request alone does not establish that the database disconnected or that data disappeared.
Success and failure can also coexist within one response. A storage record count may fail while the retention summary—information about which historical states remain protected—succeeds. There is no need to discard the successful retention information. The console marks the failed count section separately and retains the information it did obtain.
If the browser receives this response successfully, “Last success” may advance. That means the response was successfully received. It does not mean every metric inside it was collected or that the storage system's health was checked. The storage screen therefore shows both when the response was received and each section's collection result. It also keeps “integrity not checked” distinct from a healthy verdict.
A gap in a graph is part of the observation
Now suppose requests recover at 10:04 and a new sample arrives. Drawing a straight line from the 10:00 point to the 10:04 point can suggest that changes were observed throughout the interval. But if requests failed in between, that line is not a measurement. The same applies when periodic requests stop because the user switches to another browser tab.
The console's process history contains only samples obtained through successful responses. It does not fill in history from before the operator opened the page. Long gaps and process restarts begin a new segment. If server time stops or moves backward, or a cumulative counter decreases, the values on either side are not treated as a continuous change.
For example, suppose the previous sample reported 120 memory collections and the value after reconnecting is 2. Calculating a “decrease of 118 collections” is not useful. The process may have restarted; at a minimum, there is no basis for comparing the values within the same cumulative interval. The console marks the comparison as unavailable. These memory collections are Java process GC, a different operation from storage GC that removes obsolete ledger verification data.
Breaking the line makes the graph look less smooth. Yet the gap preserves the fact that no data was collected. This screen provides a short history observed during the current session. It is not a long-term operations record covering periods when the page was closed.
An accepted command is not a completed job
After requests recover, suppose the same operator checks a storage cleanup job. They start a diagnostic operation from the screen, and the server accepts the request. Marking cleanup complete merely because that response arrived would repeat the earlier mistake of presenting an old sample as current. The conclusion would exceed what the evidence establishes.
Manual GC in nigo-protocol keeps the command acceptance record separate from the current job state. The acceptance record remains evidence that a particular command was accepted. Subsequent diagnostic progress, waiting for deletion approval, deletion progress, and completion are read from the job state. Finishing a diagnostic operation does not authorize deletion, and approving deletion does not mean every candidate has been deleted.
The article on safe GC stops and actual completion explains why the distinction matters. A job can stop to preserve its protection conditions, keeping data safe without achieving the cleanup goal. Combining “accepted” or “no error” into “complete” hides that difference.
Commands whose responses never arrived also need careful interpretation. If the connection failed after the server accepted a command, the job may have started even though the browser received no response. When the outcome is uncertain, the first step is to look up the same command identifier and its job. The screen does not automatically resubmit the operation with a new identifier. Even a lookup that finds no record is not treated as proof that the command never ran.
Closing the browser is also different from asking the server to stop a job. Manual GC can reject a stop request while a batch is running. Clicking Stop does not establish that the job stopped; the operator must confirm that the current server-side job state is paused. Nor does stopping undo deletions already committed. Separating the lifetime of the screen from the lifetime of the server job makes it possible to follow the same job correctly after reconnecting.
Preserve the limits of what the screen can say
These principles require more than adding a warning to a failed screen. Each value has an observation time and a collection result. Each command has an acceptance outcome and an execution result. Gaps in a graph have reasons. Reducing these different facts to a single green “healthy” label removes evidence an operator needs to decide what to do next.
The console's regression checks address these distinctions directly. They check that other consensus observations remain available when a process refresh fails, that errors appear beside retained samples, and that one storage section's collection failure is not turned into another section's failure or an empty store. Tests of what the screen means serve a different purpose from tests of actual node and storage operations.
At 10:03, the operator needs more than a vague assurance that everything is still healthy. They need to know what was checked at 10:00, what could not be checked afterward, and which evidence to read next. Preserving those boundaries helps an operator avoid interrupting a node that is simply idle while also recognizing when a real failure needs investigation.