# Beyond Code Reasoning: Specification-Anchored Auditing of Multi-Implementation Distributed Protocols

Crypto.world

> Production blockchains stay safe by running many independently written clients of one shared specification, so a defect where one client diverges from the spec can fork the network or halt finality.

**arXiv**: 2604.26495

**Published**: 2026-04-29

**Authors**: Masato Kamba, Hirotake Murakami, Akiyoshi Sannai

## AI summary

## Overview

**Research area:** Blockchain and distributed-systems security, specifically LLM-driven security auditing of multi-client protocol implementations.

**Technical level:** Advanced. The paper assumes familiarity with byzantine-fault-tolerant consensus, Ethereum client architecture (consensus/execution split, PeerDAS/EIP-7594, KZG commitments), and LLM-agent pipelines.

**Scope (one sentence):** The paper introduces SPECA, an LLM-driven audit framework that derives explicit, typed security properties from natural-language protocol specifications and reuses them across independent client implementations, evaluated on the Sherlock Ethereum Fusaka Audit Contest (10 targets, 366 auditor submissions) and the RepoAudit C/C++ benchmark (15 projects, 35 non-disputed ground-truth bugs).

## What This Paper Is About

Production blockchains stay safe by running many independently written clients of one shared specification, so a defect where one client diverges from the spec can fork the network or halt finality. Existing LLM auditing tools reason one repository at a time and therefore have no way to express a requirement that lives in the specification but is never written into the code. SPECA closes that gap by turning the specification into a reusable set of typed security properties that are checked against each implementation under a constant baseline.

## Key Contributions

1. **Cross-implementation comparison under a shared property vocabulary.** Properties derived once from the specification are reused across 10 Ethereum audit targets — 7 consensus/execution clients, 1 EVM library, and 2 KZG cryptographic libraries spanning 6 languages — holding the security baseline constant while the implementation varies.
2. **A specification-anchored audit framework (SPECA).** A six-phase pipeline (specification discovery, subgraph extraction, property generation, code pre-resolution, property-grounded audit, severity-preserving review) that recovers all 15 in-scope H/M/L Sherlock vulnerabilities in the expert-augmented configuration (8/15 automated-only) and independently discovers 4 fix-confirmed bugs, including a cryptographic-invariant violation missed by all 366 contest submissions.
3. **Pipeline-phase-traceable failure analysis.** All analyzed false positives map to three root causes pinned to specific phases (trust-boundary misunderstanding, code-reading error, specification misinterpretation), and a multi-model study identifies property-generation quality — not model reasoning — as the binding constraint on coverage.
4. **Engineering principles for recall-first auditing.** The paper isolates the productive property types (invariants 75% precision, preconditions 100%) and characterizes a review filter that removes 30 of 102 raw findings while losing zero H/M/L true positives, at roughly $30 per H/M/L bug (about 42 minutes wall-clock under parallel execution).

## Main Findings

- **Full recall with expert augmentation, partial with automation alone.** On the Sherlock benchmark, SPECA recovers 15/15 (100%) of the 15 valid H/M/L issues (5 High, 2 Medium, 8 Low) when 7 manually authored properties are added, and 8/15 (53%) in the automated-only configuration. All 8 auto-generated properties cover ordinary code patterns; the 7 manual ones cluster in cryptographic invariants (KZG edge cases, BLS12-381 identity element) and protocol lifecycle rules.
- **Precision depends heavily on granularity.** Post-review (N=72 findings), strict H/M/L-match precision is 26.4% (19/72), confirmed-useful precision is 59.7% (43/72), and broad non-FP precision is 66.7% (48/72), with 24 confirmed false positives (33.3%). The paper attributes the strict/broad gap to a "property neighborhood effect": 11 issues are matched by two or more complementary properties (#308 and #331 each by 4, #169 by 3), which is described as genuine redundancy rather than alert duplication.
- **Cluster-level precision nearly doubles.** Grouping findings by issue yields cluster-level strict precision of 48.7%, with 15 issues carried by 19 clusters rather than 72 individual alerts.
- **Four bugs found that no contest submission reported.** All four were later confirmed by developer fix commits: the c-kzg-4844 KZG batch-verification challenge computed from the original rather than deduplicated commitment array (fix `f18ba082`); an inverted logic error plus missing validation in Lodestar (`3b98c59c`); an unchecked array access in Nimbus reachable from RPC/Engine API (`b3a3f3f9`); and a wrong subnet parameter computation plus missing cell-count validation in Prysm (`b5bdd65f`).
- **All detections are judged specification-essential.** Retrospective classification of all 15 matched issues found 15 spec-essential and 0 code-inferrable, spread across four categories: spec-defined algorithm (5 issues: #40, #176, #210, #308, #319), mathematical invariant (2: #48, #203), cross-field constraint (5: #190, #343, #371, #376, #381), and protocol parameter (3: #15, #109, #216). Two of the 15 trace all the way to a specific `INV` label (e.g., issue #319 through subgraph `SG-001` to `INV-004`).
- **One property can both detect and certify.** Property `PROP-6a4369e9-inv-014` (fork-boundary blob parameters must come from the canonical schedule) fires as a true positive on Grandine, Lighthouse and Prysm (matched to contest issue #11) while returning PASS on Nimbus, which reads the canonical schedule correctly.
- **Property-type ablation (N=88).** Invariants: N=67, 18 TP, 6 FP, 75.0% precision, driving 76% of findings. Preconditions: N=11, 4 TP, 0 FP, 100.0%. Postconditions: N=5, 1 TP, 1 FP, 50.0%. Assumptions: N=5, 0 TP, 1 FP, 0.0% — described as too noisy for reliable auditing. Excluding assumptions leaves precision essentially unchanged (67.1%) with recall stable (27.1% strict).
- **False positives trace to three pinned causes (N=16 deeply analyzed from 44 total).** Trust-boundary misunderstanding: 8 (50.0%, Phase 3). Code-reading error: 6 (37.5%, Phase 5). Specification misinterpretation: 2 (12.5%, Phase 3). The Phase 6 gate filters half the trust-boundary FPs and two-thirds of the code-reading FPs, but none of the misinterpretation ones.
- **The review filter preserves severity.** Phase 6 improves broad precision from 56.9% to 66.7% while preserving 100% recall on H/M/L, raising F1 from 72.5% to 80.0%. The 10 true positives it filters are all informational or partial (8 `tp_info`, 1 `partially_fixed`, 1 `potential-info`). Its `DISPUTED_FP` verdict has a 33.3% TP rate, meaning it acts as a false-positive detector with 66.7% precision.
- **On RepoAudit, SPECA matches the best published precision.** With Sonnet 4.5: 88.9% precision (6 FPs), 12 author-validated beyond-ground-truth candidates, cost $81.05; combined with 100% recall on the 35 non-disputed ground-truth bugs this gives F1 = 0.94, matching RepoAudit's strongest published configuration (Claude 3.7 Sonnet, F1 = 0.94). Under a partially controlled same-backbone comparison with DeepSeek R1, SPECA reports 72.7% precision with 7 new candidates while RepoAudit reports 87.2%.
- **A capability inversion on RepoAudit.** Sonnet 4 discovers more beyond-GT candidates (18) at 81.2% precision than the stronger Sonnet 4.5 (12 at 88.9%) — an effect the authors call "property adherence": stronger models stay inside the stated property scope, weaker models drift into adjacent code patterns where some genuine bugs happen to live.
- **Symmetrical disagreement with RepoAudit.** SPECA flags 5 of RepoAudit's 40 published bugs as defensive-coding fixes with no reachable exploit path; DeepSeek R1 rejects all 5 and Sonnet 4.5 rejects 4 of 5.
- **Framing matters more than prompting for bugs.** Preliminary experiments with an adversarial "find bugs" framing produced an 88% false-positive rate, which motivated the Hoare-style proof-attempt framing where a finding is emitted only when no discharging argument is reachable.
- **Cost.** $400–620 total API cost for the Sherlock evaluation (~$27–41 per recovered H/M/L bug, or $21–33 per unique bug including the 4 novel fix-confirmed bugs); $81.05 ($1.69/bug) for the RQ2 primary configuration; $93–101 per model for the multi-model sensitivity runs. Phase 5 completes in about 42 minutes wall-clock for all 10 Sherlock targets under parallel execution.
- **Prior live deployment.** An earlier version of SPECA was run live in the Sherlock contest (Sep–Oct 2025) and placed 6th overall with the largest valid-issue count (17) among participating teams. The reported analyses are from a controlled post-contest re-experiment.

## Methodology in Plain English

SPECA splits auditing into two halves: work out what the code *should* do, then check whether each implementation *does* it.

The first half runs once per specification. It crawls the specification documents, breaks them into program graphs annotated using RFC 2119 modal verbs (the "MUST"/"SHOULD" language), and applies STRIDE threat modeling with CWE Top 25 augmentation to generate typed security properties. Each property is one of four kinds: an invariant that must always hold, a precondition that must hold before an operation, a postcondition that must hold afterwards, or a trust assumption the component does not itself enforce.

The second half runs once per target repository. A Tree-sitter-based linking step maps each property to concrete source locations, which the paper says reduces downstream audit tokens by 40–60%. Then, for each property, the LLM tries to *prove* the property holds, in three steps: Map (split the property into verifiable sub-claims and find the code responsible for each), Prove (check input coverage, path coverage, concurrency safety, temporal validity, and implementation-pattern obligations such as cache keys being computed from complete inputs), and Stress-test (if the proof failed, construct a concrete attack path; findings with no plausible path are downgraded to potential-vulnerability). This is structured LLM evidence construction, not formal verification.

A final review phase may only dispute a finding through three narrow mechanical gates — dead code, trust boundary, and scope — deliberately excluding any "design intent" or "code quality" reasoning. The authors simplified this from a five-gate prototype after dropping the Spec Cross-Reference and Exploitability gates, which cost recall on informational true positives at negligible precision benefit. The whole framework is about 1,900 lines of async Python orchestration plus 1,373 lines of analysis prompts across 9 workers, with Claude Opus used for Phases 1–3 and Claude Sonnet for Phases 4–6.

## Why This Matters

**Impact on research.** The paper reframes a security-auditing result as a *representation* result: the authors argue their gains are representation-induced rather than model-capability-induced, since all 15 detections were judged impossible to infer from code alone. It also provides something rare in LLM security tooling — false positives that are traceable to a named pipeline phase rather than presented as opaque model error, converting model failure into ranked engineering targets.

**Real-world applications:**

- **Multi-client blockchain networks.** Ethereum, and by the framework's own extension Cosmos, Solana and QUIC, run heterogeneous clients of one specification; a divergence bug in any one client has network-wide blast radius rather than host-local.
- **Client onboarding and fork-readiness checks.** The `PROP-6a4369e9-inv-014` case shows a single property acting as a bug detector in deviating clients and a conformance certificate in compliant ones — useful before a scheduled hard fork.
- **Shared-library auditing.** The KZG bug lives in a cryptographic library used by several clients simultaneously, so a single missed invariant propagates across the ecosystem.
- **Triage budgeting for security teams.** With explicit per-bug costs (~$30 per H/M/L bug, ~42 minutes wall-clock) and a review filter that drops 30 of 102 findings with no H/M/L loss, teams can reason about audit economics quantitatively.

**Industry relevance.** The RepoAudit comparison places SPECA at 88.9% precision, matching the best published RepoAudit configuration, while producing 12 author-validated bugs beyond that benchmark's ground truth and beating the two industrial static analyzers reported (Meta Infer at 77.8% precision on 7 TP, Amazon CodeGuru at 0.0% on 0 TP). The paper's willingness to flag 5 of RepoAudit's 40 published bugs as defensive-coding fixes with no reachable exploit path — with model-based cross-checks (DeepSeek R1 rejecting all 5, Sonnet 4.5 rejecting 4 of 5) — illustrates a broader industry tension between pattern-based and specification-based notions of a bug.

## Future Directions

- **Automating cryptographic and protocol-lifecycle property extraction.** The 7 manual properties concentrate in KZG edge cases, the BLS12-381 identity element, custody bounds, cache-key completeness, and fork-transition refresh; the paper names improving automated extraction here as a concrete research frontier.
- **Postcondition and assumption generation.** These are the weakest property types in the ablation (50.0% and 0.0% precision) and are flagged as the specific frontiers for the automated-only configuration.
- **Reducing trust-boundary false positives.** Half of all analyzed false positives originate in Phase 3 trust-boundary misunderstanding; the Phase 6 gate catches only half of them, leaving a clear pipeline-level improvement target.
- **Extending beyond static scope.** Specifications-level flaws, hardware attacks, side-channel and supply-chain attacks, and vulnerabilities that require dynamic execution to observe (such as race conditions) are explicitly out of scope, and head-to-head comparison with individual contest participants is precluded by confidentiality — leaving a controlled baseline comparison as open work.

## Target Audience

Blockchain client and protocol security engineers, security researchers working on LLM-based or specification-driven program analysis, audit firms and bug-bounty participants who need to reason about cross-implementation conformance, and LLM-agent researchers interested in how task framing (proof-attempt versus adversarial "find bugs") and property quality — rather than raw model capability — govern detection coverage and false-positive structure. Readers without a distributed-systems background will find the evaluation accessible but the property vocabulary and the KZG/fork-boundary case studies difficult without domain context.

## Authors’ abstract

Code-driven auditing fails when correctness depends on what the specification requires rather than how the code is written. Production blockchain networks expose this directly: byzantine consensus runs many independent clients of a shared specification, so a specification-divergence defect in one client can fork the network or halt finality. Existing tools reason one repository at a time, with no shared baseline held constant across implementations. We present SPECA, an…

[The full abstract on arXiv](https://arxiv.org/abs/2604.26495)

[Read the original paper](https://arxiv.org/abs/2604.26495)

---

Canonical: https://crypto.world/paper/beyond-code-reasoning-specification-anchored-auditing-of-multi-2604.26495
