RocksDB Benchmark Methodology & Bug Analysis for paradigmxyz/reth

ROCKSDB_BENCHMARK_METHODOLOGY.md

RocksDB Benchmark Methodology & Bug Analysis

Executive Summary

Bug Found: RocksDB pending batch writes are not visible to concurrent history index reads, causing EVM execution divergence during engine_newPayload processing.

The Bug

Root Cause

When RocksDB is enabled for history indices (account_history_in_rocksdb, storages_history_in_rocksdb), there is a race condition between writes and reads:

1. Write Path: save_blocks() → write_blocks_data() → pushes to pending_batches
2. Read Path: engine_newPayload → HistoricalStateProvider → with_rocksdb_tx() → creates new transaction
3. Problem: The new read transaction only sees committed data, not pending batches

Reproduction Scenario

1. Block N is processed via Engine API
2. Block N's history indices are written to pending_batches (not yet committed)
3. Block N+1 arrives via Engine API before block N's DatabaseProvider::commit() is called
4. Block N+1 needs historical state from block N (parent's state)
5. HistoricalStateProvider looks up history indices in RocksDB
6. RocksDB returns stale data (missing block N's indices)
7. Lookup returns HistoryInfo::NotYetWritten or HistoryInfo::InPlainState incorrectly
8. EVM reads wrong state values → gas divergence

Evidence

• Block 24263292 failed with gas mismatch: expected 40,020,183, got 39,935,121 (difference: ~85,062 gas)
• This indicates one or more transactions executed with incorrect state
• The benchmark ran 901 blocks successfully before the divergence
• MDBX-only run did not have this issue (MDBX uses a single transaction with read-your-writes semantics)

Affected Code Paths

Write side (crates/storage/provider/src/providers/rocksdb/provider.rs):

// Lines 517-560: write_blocks_data pushes to pending_batches
fn write_blocks_data<N: reth_node_types::NodePrimitives>(
    &self,
    blocks: &[ExecutedBlock<N>],
    tx_nums: &[TxNumber],
    ctx: RocksDBWriteCtx,
) -> ProviderResult<()> {
    // ... writes to pending_batches, NOT committed yet
}

Read side (crates/storage/provider/src/traits/rocksdb_provider.rs):

// Lines 21-33: with_rocksdb_tx creates NEW transaction
fn with_rocksdb_tx<F, R>(&self, f: F) -> ProviderResult<R> {
    let rocksdb = self.rocksdb_provider();
    let tx = rocksdb.tx();  // NEW transaction, only sees committed data
    f(&tx)
}

Fix Options

1. Immediate commit: Commit RocksDB batches immediately in write_blocks_data instead of deferring to DatabaseProvider::commit()

   • Pro: Simple fix
   • Con: Loses atomicity with MDBX/static file commits

2. Shared transaction: Use a single RocksDB transaction per DatabaseProvider that is shared between reads and writes

   • Pro: Maintains MDBX-like semantics (read-your-writes)
   • Con: Requires refactoring RocksDB provider architecture

3. Memory overlay: Cache pending history writes in memory and check memory before RocksDB reads

   • Pro: No RocksDB changes needed
   • Con: Additional complexity and memory usage

4. Flush before read: Ensure pending batches are committed before any historical state read

   • Pro: Conceptually simple
   • Con: May introduce performance overhead and ordering complexity

Benchmark Methodology

Setup

1. Binary builds:

   • reth-rocks: Built from rocksdb-benchmark-combined branch with --features 'jemalloc,asm-keccak,rocksdb,edge'
   • reth-mdbx: Built from main with --features 'jemalloc,asm-keccak,edge'

2. Environment:

   export DATADIR=~/.local/share/reth/mainnet
   export JWT_SECRET=~/.local/share/reth/mainnet/jwt.hex

3. RocksDB storage settings (for reth-rocks):

   reth --storage.rocksdb.transaction-hash-numbers \
        --storage.rocksdb.account-history \
        --storage.rocksdb.storages-history

Benchmark Command

./reth-rocks-bench new-payload-fcu \
    --rpc-url https://reth-ethereum.ithaca.xyz/rpc \
    --engine-rpc-url http://127.0.0.1:8551 \
    --jwt-secret $JWT_SECRET \
    --from <start_block> \
    --to <end_block> \
    --output ./benchmark-results

Metrics Collected

1. newPayload latency: Time for engine_newPayloadV4 to return VALID
2. FCU latency: Time for engine_forkchoiceUpdatedV3 to return VALID
3. Ggas/s: Gigagas processed per second (gas_used / execution_time)
4. Block count: Number of blocks successfully replayed before failure

Expected vs. Observed Results

Metric	MDBX (baseline)	RocksDB
Blocks replayed	767 (connection reset)	901 (execution divergence)
Failure type	Network error	Invalid block (gas mismatch)
Execution correctness	Correct	Incorrect

Relevant Files

• crates/storage/provider/src/providers/rocksdb/provider.rs - RocksDB write/read implementation
• crates/storage/provider/src/traits/rocksdb_provider.rs - with_rocksdb_tx trait
• crates/storage/provider/src/providers/state/historical.rs - Historical state lookups
• crates/storage/provider/src/either_writer.rs - MDBX/RocksDB routing
• crates/storage/provider/src/providers/database/provider.rs - Pending batch commit logic
• bin/reth-bench/src/bench/new_payload_fcu.rs - Benchmark command

Conclusion

The RocksDB integration has a fundamental consistency issue where pending history index writes are not visible to concurrent historical state reads. This causes EVM execution to diverge when processing blocks faster than they can be committed. A fix is required before RocksDB can be used for history indices in production.