Testing Storage Contracts Before Adding RocksDB
Suppose we are receiving six state items in two batches. The first three were stored successfully. The second batch contains a duplicate, so storage rejects it. Progress still reports only three items received. But when we inspect the temporary storage area, we find five.
An exception reported the failure, and progress did not advance. What went wrong? The beginning of the failed batch had already been stored, but the progress record knew nothing about it.
This was a problem we found in nigo-protocol's existing Memory backend, which keeps state in memory, while preparing for a new storage implementation. On September 7, 2026, before implementing RocksDB support, we ran the same checks against Memory and the relational database backend. Work intended to establish a standard for the new implementation ended up fixing the implementation that was supposed to serve as its reference.
Two Items Survived a Rejected Batch
When a node catches up with another node's state, it can receive a snapshot: the complete state at a particular point in time. Instead of processing a large snapshot all at once, it receives smaller batches called chunks. The incoming data accumulates in a temporary area separate from the state currently in use. This preparation stage is called staging.
Let us call the six state items A through F, in ID order. The letters simplify the example. The actual checks compare cell IDs. We are counting state cells here, not balances.
The first chunk should be A, B, C, followed by D, E, F.
Suppose the second arrives as D, E, E instead. That input violates the requirement to store the cells in order without duplicates.
The original Memory implementation checked one item and immediately inserted it into staging if it was valid. It checked and inserted D, then checked and inserted E, before discovering the duplicate at the final E. An exception stopped the chunk from completing. Execution never reached the code that updated progress. But D and E, which had already been inserted, remained.
| Point of inspection | State items in staging | Cell count in the progress record |
|---|---|---|
| First chunk received | A, B, C | 3 |
| Before the fix, malformed second chunk rejected | A, B, C, D, E | 3 |
| After the fix, the same input rejected | A, B, C | 3 |
| After the fix, valid second chunk retried | A, B, C, D, E, F | 6 |
The active ledger state was separate from staging. This should not be read as an incident in which those five cells were immediately applied to the active ledger. The problem was that internal records disagreed while the snapshot was being received. The caller expected to be able to send the second chunk again, but storage already held part of it.
The fix separated validation from application. First, check the order and uniqueness of the entire chunk and calculate the new cell and byte counts. Only after all of that succeeds do we apply the cells and progress information. A chunk rejected during validation leaves A, B, C and their progress record intact, allowing a retry with the valid chunk.
The tests did not stop at checking that an exception was thrown. After failure, they inspected the actual cell list, the position of the next chunk, the cell and byte counts, and the active state. They also checked that submitting the correct chunk afterward completed the six-cell snapshot. The same promise applied whether a later item was a duplicate or appeared out of order.
What It Means to Share a Storage API
Memory and a relational backend are not interchangeable simply because they expose storage functions with the same names. Even if writing and reading valid input produces the same result, callers cannot retry safely if different data remains after failure.
This is where a shared contract is needed. Define the behavior a storage consumer can expect, then give each implementation the same input and check for the same result. One may use a hash map internally while another uses SQL.
For snapshots, the contract is more specific than “store a chunk.” Rejected input must not change what was already received, and progress must agree with the stored contents. The active state must remain unchanged until the complete data is ready. Receiving a previously successful request again must not duplicate data already processed.
We extended the same approach to state cell storage and queries, transactions and receipts, finalized transaction locations, state commitment trees, and snapshots. For example, we checked whether query results preserve the requested order, whether serializing a stored transaction again produces the same bytes, and whether rejecting conflicting contents leaves the original stored transaction intact.
The checks lived in the modules that own each shared interface. Memory and the relational backend ran the same scenarios, differing only in how they were set up. SQL and database connection details stayed out of the shared tests, so another storage backend could later be asked the same questions.
Returning the Same Value Was Not Enough
Another difference exposed by the shared checks concerned state history. The information linking a cell to the transaction that created it, the transactions that changed it, and the transaction that consumed it is called lineage. Even when the current value is the same, different classifications produce different accounts of the past.
When an EVM-family transaction has a cell with the same ID in both its consumed inputs and its outputs, history queries classify the relationship as a mutation of that cell. If the cell is consumed without an output carrying the same ID, its deletion must be distinguished from a mutation.
At the time, the Memory implementation included only ordinary EVM transactions in this classification. For Type-2 and Type-4 transactions, it treated replacement under the same ID as separate creation and spending relationships. Conversely, for ordinary EVM transactions, it classified spending without a replacement output as a mutation too. This differed from the EVM-family rules used by the relational backend.
The fix aligned the set of EVM-family transaction types and checked for an output with the same ID when identifying a mutation. It did not change transaction execution rules or the original receipt bytes. It corrected the different histories returned by different backends after storing the same execution result.
If we had only compared the new backend against the existing Memory implementation, we might have copied the incorrect classification as well. Establishing a shared contract required asking “What answer should the caller receive?” before asking “Does it match the existing implementation?” The existing implementation had to answer that question too.
A Shared Contract Does Not Mean Shared Durability
None of this means that running the same tests against every backend is sufficient. Memory and disk-based storage can fail at different points.
Validating an entire chunk before applying it to Memory addresses the problem of malformed input leaving a partial write. A relational backend can encounter a database error during the actual write, after input validation has passed. That requires a separate check: do the deletes and inserts already performed roll back with the entire transaction?
To test this, we deleted an existing cell and began inserting new ones, then triggered an actual database constraint error during a later insertion. After the call failed, the original cell had to be present and the new cells had to be absent, including those inserted earlier in the batch. We did not wrap the entire test in an outer transaction that would automatically roll everything back. Instead, we read the state after the production storage adapter's call had finished to verify that boundary.
We also examined block commits spanning multiple storage areas. Memory and the relational backend received the same write set, and their successful results were compared. For the relational backend, we injected failures during commit and checked that the previous state remained. We then closed and reopened the database file normally, checked that the same state persisted, and verified that a fresh attempt could succeed.
We did not extend these results into a claim that Memory's multiple stores roll back together on failure. Reopening a file is also different from forcibly terminating a process or cutting power. The work described here is the September 7 verification of Memory and the H2-based relational backend. It does not, by itself, establish the durability of RocksDB or every production database.
The article on whether a signature may be sent once the save call returns examines this distinction further. Committing its own transaction, completing synchronization to storage, and recovering after a restart are separate things to verify. We need both a shared meaning for storage calls and evidence of which failures a particular backend can withstand while keeping that promise.
Making Room for a New Backend
Before adding RocksDB, we had established more than an interface with a matching list of functions. We had turned the expected inputs, the state remaining after rejection, and the behavior of successful calls and retries into executable questions. A new implementation could answer those questions and separately verify the storage and recovery boundaries specific to it.
The fact that this process changed the existing Memory implementation matters. The shared contract became both an entry requirement for new implementations and a way to revisit an implementation we already trusted.
Returning to the snapshot example, the goal is clear. If progress says three cells, the stored prefix must actually be A, B, C. Retrying a failed chunk must not encounter cells silently left behind by that failure. After the correct chunk arrives, the actual six cells and the progress record must agree.
Even when storage provides this consistent state, an operational dashboard leaves us with another question. When was the healthy state on the screen actually observed? The accuracy of a stored fact and confidence that we are still observing it are the next things to distinguish.