Reth Performance Report — ~10M gas blocks on main (f5cf90227b) — 2026-02-16

combined_latency.csv

report.md

Reth Performance Report — ~10M Gas Blocks (mainnet) # Multiproof Chunk Size Benchmark — --engine.multiproof-chunk-size

Date: 2026-02-16 Commit: f5cf90227b (origin/main) Build: maxperf-symbols profile (jemalloc, asm-keccak, MDBX — non-edge) Machine: dev-yk (bare metal) CPU Profile: Firefox Profiler

Setup

• Snapshot: reth-1-minimal-nightly.tar.zst — mainnet MDBX snapshot at block 24,463,558
• Payloads: 345 pre-generated ExecutionPayloadEnvelopeV4 blocks (24463559–24463903), ~10M gas each
• Gas limit: 60M (16.5% utilization at ~10M gas)
• Benchmark mode: reth-bench replay-payloads with --wait-for-persistence --wait-time 400ms
• Profiling: samply (CPU), Prometheus metrics (port 9001)

Summary

Metric	Value
Blocks processed	345
Total gas	4,388,793,942 (4.39 Bgas)
Execution-only time	8.36s
Execution Ggas/s	0.5247
Wall-clock Ggas/s	0.0294 (includes 400ms wait + persistence waits)
Total transactions	44,542
Avg gas/block	12.72M
Avg txs/block	129

newPayload Latency Distribution

Percentile	Latency
Min	8.81ms
P50	20.84ms
P90	30.97ms
P95	37.78ms
P99	92.08ms
Max	205.64ms
Mean	23.70ms

Per-Block Throughput (Ggas/s)

Percentile	NP+FCU	NP Only
P5	0.289	0.293
P50	0.557	0.572
P95	1.042	1.072
Mean	0.595	0.611

From Prometheus Metrics (gas_per_second)

Percentile	Throughput
P0 (min)	0.122 Ggas/s
P50	0.593 Ggas/s
P90	0.793 Ggas/s
P95	0.852 Ggas/s
P99	0.912 Ggas/s
P100 (max)	0.988 Ggas/s

⚡ Time Breakdown (Execution vs State Root)

This is the key finding for small blocks:

Component	Total Time	% of NP	Avg/block
EVM Execution	5.068s	68.4%	14.69ms
State Root	2.343s	31.6%	6.79ms
Pre-execution overhead	0.047s	0.6%	0.14ms
Post-execution overhead	0.018s	0.2%	0.05ms
Tx wait (scheduling)	0.064s	0.9%	—
Total	~7.41s	100%	~21.5ms

Execution Detail

Metric	Value
Total tx execution	4.814s (44,542 txs)
Avg tx execution	0.108ms/tx
Avg block execution	14.69ms

State Root Percentiles

Percentile	Duration
P50	6.85ms
P90	11.33ms
P95	11.95ms
P99	13.46ms
Max	57.32ms

Execution Percentiles

Percentile	Duration
P50	10.32ms
P90	17.14ms
P95	23.55ms
P99	32.02ms
Max	56.43ms

Trie/State Root Internals

Metric	Value
Proof task account worker idle time (P50)	2.0ms
Proof task account worker idle time (P90)	11.2ms
Proof task account worker idle time (P99)	28.8ms
Proof task storage worker idle time (total)	94.07s
Deferred encoder sync operations	982,874
Deferred encoder dispatched	84,102
Trie cursor operations (state seek/next)	21,390 each
Accounts updated	69,848 (avg 202/block)
Storage slots updated	82,009 (avg 238/block)

FCU Latency

FCU is very fast — negligible compared to newPayload:

Metric	Value
Mean	0.54ms
Median	0.53ms
Max	0.86ms

Key Observations

1. Execution dominates at 68% of NP time

On these ~10M gas blocks, EVM execution takes 68.4% of newPayload time while state root takes 31.6%. This is different from the pattern we saw on very small blocks (<5M gas) where state root dominated.

2. State root is well-optimized for this workload

• P50 state root of 6.85ms is good for blocks touching ~200 accounts / ~240 storage slots
• The parallel state root task (reth_trie_state_root_task) is working effectively
• Proof task worker idle times (P50: 2ms) suggest good parallelism utilization

3. P99 tail latency driven by outlier blocks

• P99 NP latency is 92ms vs P50 of 21ms — a 4.4x ratio
• The max (205ms) suggests occasional blocks with unusual state access patterns
• These outliers are worth investigating individually

4. Transaction scheduling overhead is minimal

• Tx wait time (0.064s total) is <1% of execution — the processor spawn is not a bottleneck

Bottleneck Analysis

For ~10M gas blocks on mainnet (current state on main):	
  NP latency breakdown (P50 = 20.84ms):	
  ┌───────────────────────────────────────────┐	
  │ EVM Execution        │ 10.32ms │  49.5%   │	
  │ State Root           │  6.85ms │  32.9%   │	
  │ Overhead/scheduling  │  3.67ms │  17.6%   │	
  └───────────────────────────────────────────┘	
  At P95 (37.78ms):	
  ┌───────────────────────────────────────────┐	
  │ EVM Execution        │ 23.55ms │  62.3%   │	
  │ State Root           │ 11.95ms │  31.6%   │	
  │ Overhead/scheduling  │  2.28ms │   6.0%   │	
  └───────────────────────────────────────────┘	

Recommendations

High Impact

1. EVM execution optimization — At 68% of time, this is the biggest lever. Profile individual opcode hotspots in the EVM execution path. The 0.108ms/tx average suggests room for improvement in state access patterns during execution.

2. Investigate P99 outliers — The 92ms P99 and 206ms max are 4-10x above median. Understanding what makes these blocks slow could help improve tail latency.

Medium Impact

3. State root parallelism — While P50 is good at 6.85ms, the proof task worker idle times (P90: 11.2ms, P99: 28.8ms) suggest some work distribution imbalance. The storage worker idle time (94s total across all workers) indicates potential for better work balancing.

4. Reduce overhead gap — There's ~3.7ms of unaccounted time at P50 between execution+state_root and total NP latency. This could be FCU-related overhead, notification, or other post-execution work. ## v2 Results (with warmup — fair comparison)

Lower Priority for This Workload

5. Persistence — Not a bottleneck for ~10M gas blocks with --wait-for-persistence
6. FCU latency — Already sub-1ms, negligible
7. Transaction scheduling — <1% overhead, well-optimized

CSV Data Methodology: Each chunk size gets a full warmup pass (345-block replay to warm OS page cache), then unwind + measured pass. This eliminates cold-cache bias from v1.

Results are saved at: Setup: 345 blocks (24463559–24463903), ~10M gas each, MDBX (non-edge), commit f5cf90227b, jemalloc+asm-keccak

• /home/ubuntu/bench-results/combined_latency.csv — Per-block NP/FCU latency
• /home/ubuntu/bench-results/total_gas.csv — Per-block gas over time ### Execution Ggas/s (newPayload + FCU)
• /home/ubuntu/bench-results/metrics_before.txt — Pre-bench Prometheus metrics
• /home/ubuntu/bench-results/metrics_after.txt — Post-bench Prometheus metrics | Chunk Size | Exec Ggas/s | vs Default (60) |
• /home/ubuntu/bench-results/reth-main-profile.json.gz — samply CPU profile (2.6MB) |------------|-------------|-----------------|
• Firefox Profiler link | 15 | 1.1968 | +1.2% ⬆️ | | 30 | 1.1969 | +1.2% ⬆️ | | 60 ⭐ | 1.1828 | +0.0% ➡️ | | 120 | 1.1507 | -2.7% ⬇️ | | 240 | 1.0647 | -10.0% ⬇️ |

newPayload Latency (ms)

Chunk Size	Mean	P50	P95	P99
15	10.32	9.73	14.87	17.62
30	10.33	9.77	14.97	18.34
60	10.44	9.81	14.79	17.35
120	10.76	10.15	14.84	16.99
240	11.64	11.23	15.18	17.82

Total Latency (newPayload + FCU, ms)

Chunk Size	Mean	P50	P95	P99
15	10.63	10.02	15.20	18.00
30	10.63	10.04	15.28	18.72
60	10.76	10.08	15.16	17.72
120	11.06	10.46	15.13	17.33
240	11.95	11.56	15.51	18.15

Key Findings

1. Clear monotonic trend: smaller chunks → better throughput (30 > 15 > 60 > 120 > 240)
2. Best: chunk=15 and chunk=30 are essentially tied at ~1.197 Ggas/s (+1.2% over default)
3. Worst: chunk=240 at 1.0647 Ggas/s (−10% vs default)
4. Default (60) is reasonable but 15–30 offer marginal improvement
5. Larger chunks hurt: 120 (−2.7%) and 240 (−10%) show clear degradation
6. v1 spread was mostly cold-cache artifact: v1 showed 2.4x spread (0.52–1.21), v2 shows only 12% spread (1.06–1.20)

Recommendation

The default of 60 is fine. Chunk sizes 15–30 offer ~1% improvement but the difference is within noise. Avoid chunk sizes ≥120 which show measurable degradation. The key finding is that cold OS page cache was the primary variable in v1, not chunk size.

total_gas.csv