Research
SPECA: Specification-to-Checklist Agentic Auditing for Multi-Implementation Systems -- A Case Study on Ethereum Clients
Multi-implementation systems such as Ethereum have many independent clients that all claim to implement the same written specification, and auditors must check each one.

- arXiv
- 2602.07513
- Published
- 2026-02-07
- Authors
- Masato Kamba, Akiyoshi Sannai
AI summary
Overview
Research area: Blockchain systems security — specifically, agentic (LLM-driven) auditing of natural-language protocol specifications against multiple independent implementations, with Ethereum consensus (CL) and execution (EL) clients as the case study.
Technical level: Intermediate. The paper is readable without deep cryptography background, but familiarity with Ethereum client architecture, specification documents (EIPs), and basic security-audit concepts helps.
Scope: The paper presents SPECA, a framework that converts normative specification text into property-based checklists, maps them to code locations, and reuses each checklist item across many implementations ("1 → N reuse"), evaluated in a real Ethereum Fusaka audit contest covering 11 production clients.
What This Paper Is About
Multi-implementation systems such as Ethereum have many independent clients that all claim to implement the same written specification, and auditors must check each one. Existing automation, especially differential testing, only finds bugs when implementations disagree — so when every client makes the same wrong reading of an ambiguous requirement, the bug stays invisible (the authors call this a "semantic blind spot"). SPECA addresses this by grounding the audit in the specification itself: it extracts normative requirements, turns them into checklists, and then sweeps the same checklist item across all clients so that one human insight scales to many implementations.
Key Contributions
-
SPECA framework. An artifact-centric, two-phase pipeline (knowledge structuring, then systematic auditing) that produces inspectable artifacts: extracted normative requirements with traceability IDs, specification-to-implementation maps, property-based checklists, and audit reports with proof traces.
-
Cross-implementation checklist reuse as a first-class strategy. The paper operationalizes "1 → N reuse": once a checklist item fails in one client, an abstracted pattern is searched for in all other clients, exploiting the correlation structure of shared misinterpretations rather than relying on behavioral divergence.
-
In-the-wild case study. A deployment in the Ethereum Fusaka audit contest (11 production clients), with 54 submissions, 17 judged valid (31.5%), and cross-implementation checks accounting for 76.5% (13/17) of valid findings.
-
Failure-mode characterization and a systems-security lesson. Manual coding of the 37 invalid submissions shows threat model misalignment explains 56.8% (21/37) of false positives, and analysis of the 7 missed High/Medium issues identifies concentrated gaps that motivate the V2 redesign (explicit threat-model formalization).
Main Findings
-
Cross-implementation checks dominate valid findings: 13 of 17 valid findings (76.5%) came from cross-implementation checks; specification-based static audit contributed 3 (17.6%), and dynamic test generation 1 (5.9%).
-
Valid rate above contest average: SPECA's 54 submissions yielded 17 valid (31.5%), versus the contest-wide 101/366 (27.6%). Client coverage was 9 of 11 clients (81.8%).
-
But no High/Medium findings in V1: All 17 valid findings were Low (1) or Informational (16). In the contest there were 5 High and 2 Medium issues; V1 found 0% of them, and 1 of 8 Low issues (12.5%).
-
False positives are dominated by threat-model misalignment: Of 37 invalid submissions, 21 (56.8%) were threat model misalignment, 8 (21.6%) already known/duplicate, 5 (13.5%) incorrect analysis, and 3 (8.1%) out of scope. The prompt assumed EL responses were untrusted, while contest rules treated the EL as trusted.
-
Misses cluster in three categories: Of the 7 missed High/Medium issues, 4 (57.1%) were specification details and implicit assumptions (judged addressable via finer-grained checklists), 2 (28.6%) timing and concurrency issues (judged a current LLM limitation), and 1 (14.3%) an external library dependency (the
c-kzg-4844library, addressable via supply-chain scanning). -
V2 improves high-impact detection: After adding explicit threat-model formalization (actor trust levels TRUSTED, SEMI_TRUSTED, UNTRUSTED, plus boundary edges), a post-hoc re-evaluation on the 11 CL-client issues rated Low/Medium/High achieved 27.3% strict recall (3/11) — 2 of 3 High (66.7%), 0 of 2 Medium (0.0%), 1 of 6 Low (16.7%).
-
The matched issues: the proposer-lookahead miscalculation (High), the weak Fiat-Shamir weakness in
c-kzg-4844verify_cell_kzg_proof_batch(High), and Lodestar accepting/rebroadcasting data column sidecars with invalid signatures (Low). -
Agent versus humans: The paper reports the V2 agent's performance places it in the top 4% of human auditors, outperforming 49 of 51 contestants in detecting critical issues.
-
Human effort drops sharply: A specialist needed about 10 minutes on average to validate a core finding, plus about 30 more minutes to refine and finalize the PoC and report — roughly 40 minutes per submission on average.
-
A hypothesis about why reuse works: The authors suggest that widespread use of a few powerful LLMs in development may create homologous bugs across independent teams, making 1 → N reuse an increasingly valuable audit strategy.
Methodology in Plain English
The pipeline has two halves.
Phase 1 — knowledge structuring. The system reads specification documents (e.g., EIP markdown) and pulls out normative statements identified by modal verbs (MUST, SHOULD, MAY, SHALL) following RFC 2119 conventions, tagging each with a unique ID (for example EIP7594-CUSTODY-001) and its source location. It also builds a database of 50+ known vulnerability patterns (boundary validation failures, guard condition omissions, state transition errors, cryptographic misuse, resource management issues) and then maps each requirement to candidate code locations using keyword search to prune, followed by LLM-based semantic refinement.
Phase 2 — systematic auditing. Three strategies run against that shared checklist: (A) specification-based static audit, checking presence, correctness, and completeness of the code implementing each requirement; (B) cross-implementation checks, which abstract a generalizable pattern from one finding, locate analogous sites in the other clients, check each, and report for human verification; and (C) dynamic test generation for boundary conditions.
V2 adds a step before auditing: the threat model is written down as a structured artifact of actor trust levels and boundary edges, so findings that require capabilities outside that model can be filtered out.
Attribution of each valid report to a strategy used the submission's own recorded discovery method (similar-issue-based → cross-implementation; code analysis → static audit; fuzz testing → dynamic). Invalid submissions were manually coded into mutually exclusive categories. The V2 re-evaluation compared agent findings against the contest's ground-truth database using strict matching (same bug class, code region, and root cause), with matching performed by an LLM-based semantic comparison (GPT-4.1-mini) that explicitly excluded partial or thematically similar overlaps. All findings still require human verification, and the specification documents are assumed correct.
Why This Matters
Impact on research. The paper reframes a known weakness of divergence-based oracles — correlated misinterpretation, going back to Knight and Leveson's 1986 N-version programming study — as something addressable with specification-anchored checklists rather than formal verification, which the authors argue is too costly for rapidly evolving specs. It also argues that false-positive root cause analysis, not just model capability, is the lever that matters: over half the noise came from an unstated trust assumption.
Real-world applications.
- Security audits of multi-client blockchain networks, where the same requirement must be checked in a dozen codebases at once.
- Protocol and standards compliance work (the paper positions RFC-style documents generally, citing RFCAudit, AUTOSPEC, and RFC-TCP/IP gap-detection work as adjacent lines).
- Audit-contest and bug-bounty triage, where reducing invalid submissions directly reduces wasted judge and auditor time.
- Supply-chain and dependency review for consensus-critical libraries, identified as an addressable miss category.
Industry relevance. The contest setting is production-scale: a 4-week contest with a $2,000,000 prize pool, 11 clients (Nimbus, Lighthouse, Prysm, Teku, Lodestar, Grandine on the consensus layer; Geth, Reth, Erigon, Nethermind, Besu on the execution layer), and severity thresholds defined by network impact (Critical > 50%, High > 33%, Medium > 5%, Low > 0.01%), with non-informational findings requiring an executable PoC at submission. A ~40-minute human cost per finalized submission is a concrete operational number for teams considering agent-assisted audits.
Future Directions
- Replicate beyond Ethereum. The authors state results come from a single deployment and that replication across other multi-implementation domains is needed to establish robustness and threat-model alignment.
- Raise checklist granularity. The largest addressable miss category (specification details and implicit assumptions, 57.1% of missed High/Medium issues) is attributed to checklists being too coarse — e.g., missing exact cache-key fields or nuanced input-validation rules.
- Integrate complementary tooling. The paper proposes adding dependency/supply-chain scanning plus fuzzing, model checking, or symbolic execution to cover external-library and timing/concurrency misses; the single external-library miss (
c-kzg-4844) is cited as directly tool-addressable. - Advance stateful reasoning. Timing and concurrency misses (28.6%) — such as metadata staleness during fork transitions, caching logic, and sync state machines — are framed as a frontier limitation of current LLM agents rather than a methodology gap, and remain an open research question.
Target Audience
Security researchers and practitioners working on multi-implementation systems; blockchain protocol engineers and client teams who audit against EIPs or similar specs; auditors and audit-contest organizers interested in agent-assisted workflows and false-positive reduction; and LLM-agent researchers looking for a rigorously measured, in-the-wild evaluation of an agentic pipeline rather than a benchmark-only result. Readers seeking a purely cryptographic or formal-verification treatment will find those aspects out of scope.
Authors’ abstract
Multi-implementation systems are increasingly audited against natural-language specifications. Differential testing scales well when implementations disagree, but it provides little signal when all implementations converge on the same incorrect interpretation of an ambiguous requirement. We present SPECA, a Specification-to-Checklist Auditing framework that turns normative requirements into checklists, maps them to implementation locations, and supports cross-implementation…
The full abstract on arXiv