Skip to content

Instantly share code, notes, and snippets.

@denniswon

denniswon/gist:d77b46c991de530b554f8c2f6d840488

Created March 8, 2026 00:17
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denniswon/d77b46c991de530b554f8c2f6d840488 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denniswon/d77b46c991de530b554f8c2f6d840488 to your computer and use it in GitHub Desktop.
Download ZIP
E2E Test Coverage Comparison
Raw
gistfile1.txt
---
E2E Test Coverage Comparison
Test Matrix
┌───────────┬──────────────────────────┬──────────────────────────┬───────────────────────────────────────────┬────────────────────┐
│ Feature / │ gateway_e2e │ dual-anvil-multichain-e2 │ multichain-routing-e2e │ privacy_e2e │
│ Scenario │ │ e │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Gateway │ │ │ │ │
│ construct │ Gateway::new() │ Gateway::new() │ Gateway::from_multi_chain_config() │ Gateway::new() │
│ ion │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Config │ NewtonAvsConfigBuilder │ NewtonAvsConfigBuilder │ MultiChainConfig::from_network(NetworkMod │ NewtonAvsConfigBui │
│ path │ (single chain) │ (single chain, dest) │ e::Local) │ lder (single │
│ │ │ │ │ chain) │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Chains │ 1 (31337) │ 1 (31338 dest, with │ 2 (31337 + 31338 simultaneously) │ 1 (31337) │
│ tested │ │ source 31337 infra) │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Chain │ N/A │ N/A │ resolve_chain_for_intent() │ N/A │
│ routing │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Per-chain │ │ No (single, dest chain │ Yes (2 independent ChainService │ │
│ ChainSer │ No (single) │ only) │ instances) │ No (single) │
│ vice │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Operator │ │ Source + cross-chain │ │ │
│ registrat │ Source chain only │ table sync │ Source + cross-chain table sync │ Source chain only │
│ ion │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Consensus │ Centralized OR Two-phase │ Centralized OR Two-phase │ Centralized only │ Centralized only │
│ modes │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Attestati │ │ │ │ │
│ on valida │ Direct / Regular / Skip │ Direct / Regular / Skip │ None │ None │
│ tion │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ On-chain │ Yes (via handle_attestat │ Yes (via handle_attestat │ │ Yes (verify_task_o │
│ task veri │ ion_validation) │ ion_validation) │ Yes (verify_task_on_chain per chain) │ n_chain) │
│ fication │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Indexer │ Yes (optimistic write │ Yes (optimistic write │ No │ No │
│ DB write │ verification) │ verification) │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ HPKE │ No │ No │ When privacy=true │ Yes │
│ encrypt │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ upload │ No │ No │ When privacy=true │ Yes │
│ encrypted │ │ │ │ │
│ data │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ dual │ No │ No │ When privacy=true │ Yes │
│ Ed25519 │ │ │ │ │
│ sigs │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ gateway d │ No │ No │ When privacy=true │ Yes │
│ ecryption │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ public │ No │ No │ When privacy=true │ Yes │
│ key verif │ │ │ │ │
│ ication │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Privacy: │ │ │ │ │
│ crypto │ No │ No │ When privacy=true │ Yes │
│ smoke │ │ │ │ │
│ test │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Multichai │ │ │ │ │
│ n infra │ No │ Yes (setup_multichain) │ Yes (setup_multichain) │ No │
│ setup │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ Task subm │ 1 task │ 1 task (dest chain) │ 2 tasks (1 per chain) │ 1 task │
│ ission │ │ │ │ │
├───────────┼──────────────────────────┼──────────────────────────┼───────────────────────────────────────────┼────────────────────┤
│ BLS aggre │ Yes │ Yes │ Yes (per chain) │ Yes │
│ gation │ │ │ │ │
└───────────┴──────────────────────────┴──────────────────────────┴───────────────────────────────────────────┴────────────────────┘
Key Findings
1. gateway_e2e vs multichain-routing-e2e — NOT redundant
gateway_e2e exercises capabilities that multichain-routing-e2e does NOT:
- Attestation validation (Direct via validateAttestationDirect, Regular via executeIntent) — this is the primary differentiator.
multichain-routing-e2e does zero attestation validation.
- Two-phase consensus — multichain-routing-e2e hardcodes centralized mode only.
- Indexer DB optimistic writes — verified in gateway_e2e, not in multichain-routing-e2e.
- Gateway::new() path — tests the standard single-chain construction path.
2. dual-anvil-multichain-e2e vs multichain-routing-e2e — Partially overlapping, but distinct
dual-anvil-multichain-e2e is run_gateway_e2e() with multichain_infrastructure: true, targeting the destination chain (31338). It tests:
- Single gateway serving ONE chain (dest), with source chain as an infra dependency
- Attestation validation on the destination chain
- Two-phase consensus on destination chain
- Indexer DB writes
multichain-routing-e2e tests a fundamentally different architecture:
- Single gateway serving TWO chains simultaneously via Gateway::from_multi_chain_config()
- Intent-based routing (resolve_chain_for_intent)
- Per-chain ChainService isolation
- Tasks on BOTH chains verified independently
These test different code paths. dual-anvil-multichain-e2e tests "can the gateway handle a cross-chain task on a destination chain?"
while multichain-routing-e2e tests "can a single gateway route tasks to multiple chains?"
3. privacy_e2e vs multichain-routing-e2e privacy=true — Partially overlapping
Both test the same 7-step privacy pipeline (encrypt → upload → dual-sig → task create → decrypt → BLS aggregate → on-chain verify). The
differences:
- privacy_e2e: single-chain (31337), single-chain gateway (Gateway::new()), verifies gateway public key against local derivation from
ECDSA key
- multichain-routing-e2e privacy=true: dual-chain, Gateway::from_multi_chain_config(), tests chain_id-scoped DB lookups for
encrypted_data_refs, uploads per-chain
The multichain variant exercises the production-mode gateway path with privacy, which privacy_e2e does not.
Coverage Gaps in multichain-routing-e2e
┌─────────────────────────────────────────┬────────────────────────────────────────────────────────────────┐
│ Missing from multichain-routing-e2e │ Why it matters │
├─────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
│ Attestation validation (Direct/Regular) │ Core security path — validates BLS proofs are correct on-chain │
├─────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
│ Two-phase consensus │ Production consensus mode for live data │
├─────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
│ Indexer DB verification │ Confirms optimistic writes work in multichain mode │
├─────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
│ chain_watcher_redis_url integration │ Not tested in any routing test │
└─────────────────────────────────────────┴────────────────────────────────────────────────────────────────┘
★ Insight ─────────────────────────────────────
1. None of these tests are fully redundant. Each exercises a distinct combination of gateway construction path, chain topology, and
validation features. The multichain-routing-e2e is the only test that uses Gateway::from_multi_chain_config() (the production default),
but it lacks attestation validation and two-phase consensus coverage.
2. The biggest gap is attestation validation in multichain mode. gateway_e2e and dual-anvil-multichain-e2e validate attestations, but
only through the single-chain gateway path. No test validates attestations when the gateway serves multiple chains simultaneously.
3. privacy_e2e remains valuable as a focused single-chain test — it verifies the HPKE public key derivation against the ECDSA key chain,
which multichain-routing-e2e privacy=true does not (it just fetches and logs the key without asserting against local derivation).
─────────────────────────────────────────────────
@denniswon
Copy link
Author

denniswon commented Mar 8, 2026

save this to local memory and /super-save supermemory for our E2E Test Coverage Comparison in newton-prover-avs, our
detailed plan that we drafted above, execution state/next TODOs, remaining items, current issues and limitations, etc.

⏺ Let me save to both local memory and supermemory in parallel.

⏺ Skill(claude-supermemory:super-save)
⎿  Successfully loaded skill · 1 tool allowed

⏺ Bash(node "/Users/denniswon/.claude/plugins/cache/supermemory-plugins/claude-supermemory/0.0.1/scripts/save-project-memory.cjs"
"[SAVE:denniswon:2026-03-08]…)
⎿  Project knowledge saved: newton-prover-avs
ID: hm466ZCdEGYCa3BJfVhNWP

⏺ Saved to both locations:

  • Local memory: e2e-unification.md with full plan + MEMORY.md updated with architectural insight
  • Supermemory: Project knowledge ID hm466ZCdEGYCa3BJfVhNWP — searchable across sessions via /super-search newton e2e test unification

★ Insight ─────────────────────────────────────

  1. The memory is now structured so future sessions can pick up Phase 1A immediately — MEMORY.md links to e2e-unification.md for the
    detailed plan, and Supermemory has the full context for cross-session search.
  2. The key decision captured: Gateway::new() is legacy, MultiChainConfig::from_network() is production. All new test work should build on
    multichain-routing-e2e as the canonical test runner.
    ─────────────────────────────────────────────────

Summary of what's saved:

┌─────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Store │ Contents │
├─────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ memory/MEMORY.md │ "E2E Test Architecture" section — production alignment insight, legacy vs production gateway │
│ │ paths │
├─────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ memory/e2e-unification.md │ Full plan: coverage matrix, 3-phase architecture, implementation order, which tests stay │
│ │ separate │
├─────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Supermemory │ Complete snapshot: coverage comparison, plan, execution state, key files, current issues │
│ (hm466ZCdEGYCa3BJfVhNWP) │ │
└─────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘

Next TODOs (in priority order):

  1. Phase 1A: Add two_phase_consensus, validate_attestation, indexer_database_url to multichain-routing-e2e
  2. Runtime test: make multichain-routing-e2e privacy=true with dual anvil
  3. PR #421: Awaiting review (title/body updated to /pr style)
  4. NEWT-179: Developer integration docs (Backlog)

Ready to start Phase 1A implementation when you give the go-ahead.

@denniswon
Copy link
Author

denniswon commented Mar 8, 2026

Phase 1A implementation is complete. Here's a summary of all changes:

★ Insight ─────────────────────────────────────
What changed and why:

  1. integration-tests/src/multichain_routing.rs — Added two_phase_consensus: bool and validate_attestation: AttestationValidationMode to
    MultichainRoutingConfig. The hardcoded false for two-phase consensus now reads from config. submit_and_verify_task returns
    TaskRequestResult (not just TaskId) so attestation validation can access the full task/response/signature data. After the source chain
    task, attestation validation runs via the existing handle_attestation_validation from gateway.rs, using to_single_chain_config() to
    bridge from MultiChainConfig to the NewtonAvsConfig that attestation functions expect.
  2. integration-tests/tests/multichain_routing.rs — Parses TWO_PHASE_CONSENSUS and VALIDATE_ATTESTATION env vars (same pattern as
    gateway.rs test harness).
  3. integration-tests/src/gateway.rs — handle_attestation_validation was already made pub in the previous session.
  4. Makefile — Passes TWO_PHASE_CONSENSUS and VALIDATE_ATTESTATION env vars through to the test binary. Default:
    two_phase_consensus=false, validate_attestation=skip (preserving backward compatibility).
  5. .claude/rules/testing.md — Updated docs with new command examples.

Design choice — attestation on source chain only: Attestation validation uses validateAttestationDirect which calls
BLSSignatureChecker.checkSignatures(). This only works on the source chain where EigenLayer's BLS registry lives. Destination chains use
BN254CertificateVerifier instead, so attestation validation is intentionally skoped to the source chain task.

Default validate_attestation=skip: This preserves backward compatibility — existing make multichain-routing-e2e invocations behave
identically. Users opt in via validate_attestation=direct or validate_attestation=regular.
─────────────────────────────────────────────────

Files changed:

  • integration-tests/src/multichain_routing.rs — config fields, consensus wiring, attestation validation, return type
  • integration-tests/tests/multichain_routing.rs — env var parsing
  • Makefile — env var passthrough
  • .claude/rules/testing.md — docs

Next steps:

  • Runtime verification: make multichain-routing-e2e validate_attestation=direct (requires dual anvil)
  • Runtime verification: make multichain-routing-e2e two_phase_consensus=true (requires dual anvil)
  • Phase 2: Unified E2ETestConfig with scenario presets (separate PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment