Research
Beyond Copying: Lifelong Smart Contract Security Task Construction via Protocol Mutation
Existing smart contract security benchmarks can be "solved" by agents that recall memorized answers or copy attack code straight from public blockchain transactions, reports, and reproduction…

- arXiv
- 2610.32453
- Published
- 2026-10-23
- Authors
- Zifan Peng, Peng Huang, Jingyi Zheng, Yingjie Xue, Xuechao Wang, Xinlei He, Xia Hu, Dongrui Liu
AI summary
Beyond Copying: Lifelong Smart Contract Security Task Construction via Protocol MutationOverview
Research area: AI agent evaluation for smart contract and blockchain security (LLM-based agents, benchmark construction, mutation testing), submitted to arXiv under Cryptography and Security (cs.CR), arXiv:2610.32453v1, 23 October 2026.
Technical level: Advanced. The paper assumes familiarity with smart contract execution, vulnerability classes, attack transactions, and LLM agent benchmarking, though its central idea is explainable in plain terms.
Scope: The paper proposes ConstraintDial, an automated framework that builds and continuously refreshes detect/exploit/patch security tasks by mutating the protocol conditions an attack must satisfy, and uses it to construct the ReCast benchmark of 3,045 task inputs from 203 public cases.
What This Paper Is About
Existing smart contract security benchmarks can be "solved" by agents that recall memorized answers or copy attack code straight from public blockchain transactions, reports, and reproduction repositories. Building and validating these benchmarks also depends on costly manual annotation, so they are hard to scale or refresh as new incidents appear.
The paper's goal is to make the answer to a security task stop working while the underlying vulnerability stays real. It does this by changing the protocol conditions needed to reach a vulnerable operation, so the attack's calls or inputs must be adjusted, while the vulnerability, the security objective, and normal protocol behavior are preserved.
Key Contributions
- ConstraintDial framework. An automated framework for lifelong construction of smart contract security tasks via protocol mutation, where attack execution is used to select changes to the calls or inputs an attack requires, and execution checks guide revisions until the original attack fails while an adapted attack still succeeds.
- The ReCast benchmark. 203 public cases with 5 protocol versions each (Base, Equivalent control, Change 1, Change 2, Combined changes), yielding 3,045 task inputs total: 1,015 each for detect, exploit, and patch. Cases are collected through DeFiHackLabs, BlockSec, and Web3Bugs, after filtering that removed cases shared with EVMbench, SCONE, and CyberChainBench.
- Quantified leakage reduction. Compared with reconstruction alone, protocol mutation reduces success with supplied public answers by 10.9 percentage points on average across detect, exploit, and patch, and the fraction of tasks solved by copying online answers falls by 9.3 points.
- An evaluation of 6 LLM-based agents on ReCast with open network access, plus construction ablations showing that using failed tests to guide revisions matters more than applying the same checks only at the end (100 accepted versions at 82.0% for the full method versus 61 at 50.0% without test feedback).
Main Findings
- Attacks break, diagnoses often survive. With a historical public answer supplied, exploit success decreases from 85.2% on Base to 72.1% on Combined changes, while detect gains from the supplied answer fall from 38.5 to 27.0 percentage points and patch gains remain 21.3 points in both versions. After mutation, 63 of 89 correct detect reports retained the supplied diagnosis and 57 of 93 successful patches retained the supplied repair logic, because those answers remain correct for the current protocol.
- Mechanical copying fails after mutation. Without an agent revising them, mechanically transferred public attacks succeeded on 5 of 10 Base tasks but only 1 of 10 Combined changes tasks; on Combined changes, 7 reports and 6 patches remained correct.
- Copying falls even without supplied answers. Across detect, exploit, and patch, the fraction of tasks solved by copying online answers dropped from 23.2% on Base to 13.9% after mutation. On mutated exploit tasks agents found public answers in 81 sessions and copied them in 55; 31 of those 55 succeeded, including sessions that revised the copied answer.
- Agents adapt rather than merely prepend calls. With protocol mutation, 60 of 88 successful attacks (68.2%) changed calls or inputs beyond adding preparation calls, versus 9 of 100 (9.0%) under simple preparation.
- Performance ranges across six models. Detect report correctness ranged from 26.6% to 46.6%, exploit success from 27.9% to 65.2%, and joint patch success from 32.5% to 62.3%. DeepSeek-V4.1-Flash achieved the highest overall success rate for each task type, at 46.6% detect, 65.2% exploit, and 62.3% patch.
- Patch scores can overstate repair. Patch attack blocking exceeded joint success by 10.2–13.4 percentage points, so blocking alone is a misleading metric.
- Mutation does not uniformly hurt. Averaged over six models, patch success was 50.5% on Base, 44.3% across Change 1 and Change 2, and 42.3% on Combined changes; exploit success rose for four models and fell for two. Qwen3.8-Max gained ten cases but lost three from Base to Combined changes, while DeepSeek-V4.1-Flash gained 12 but lost six, and all 95% confidence intervals for these changes include zero.
- Case-level reversals exist. In the SMOOFSStaking example, DeepSeek-V4.1-Flash identified the vulnerability but failed its own deposit authorization, leaving all 45,000 reward tokens in the vault, while Qwen3.8-Max corrected its callback check and left eight tokens, passing all four final checks.
- More time helps exploit most. Comparing 30-minute and 60-minute sessions on 305 tasks per task type, the extra 30 minutes improved exploit success more than detect or patch for four of six models; DeepSeek-V4.1-Flash gained 1.3–3.0 percentage points across the three tasks.
- Cost tracks differently by task. DeepSeek-V4.1-Flash had the highest detect and patch success with mean costs of $0.03 and $0.22; on exploit, GLM-5.3-Flash cost less ($0.08 versus $0.21) but succeeded less often (42.6% versus 65.2%).
- Execution feedback is the biggest construction lever. The full method accepted 100 of 122 proposals (82.0%), versus 65 (53.3%) for direct rewriting, 73 without execution traces, 75 without paired checks, and 61 without test feedback. Simple preparation accepted 105 (86.1%), but most of its successful attacks only added preparation calls. Including failed proposals, construction cost $1.50 per accepted version for the full method and $1.73 for direct rewriting.
Methodology in Plain English
The team starts from a public incident or audit finding and rebuilds a self-contained local version of the affected protocol, replacing deployed addresses and cutting dependencies that a local chain cannot provide. Each case becomes a pair of protocols: one that keeps the vulnerability and one that repairs it, both sharing the same roles, assets, initialization, and interfaces.
They then run a reference attack against the vulnerable version and watch which earlier conditions the protocol checks before the vulnerable operation can be reached, such as a registration step or a time window that must be open. From that record they select mutations that change one of those conditions in both protocols, so the same requirement now applies but is satisfied differently.
A mutation is only accepted if several things hold: the original attack, with only addresses and interface bindings updated, now fails at the changed condition; an adapted attack still achieves the fixed security objective on the vulnerable protocol and fails on the repaired one; undoing the mutation restores the original attack's success; and normal operation tests still pass on both protocols. A separate equivalent control changes code outside the attack path, leaving the original attack working, so the effect of real mutations can be isolated.
Failed checks produce evidence-rich records that drive revisions to the contracts and reference answers, capped by budgets (10 revisions in the reported experiments). Only tasks passing every protocol and task check enter the dataset, and new incidents or audit findings go through the same procedure, which is what makes the construction lifelong.
For the three task types, the same protocol pair supplies one detect input (with a reference report scored on location via a rubric, root cause, impact, and separately scored trigger coverage), one exploit input (graded on reaching the fixed objective from a fresh initialization), and one patch input (graded on building, blocking validated attacks, and preserving normal operations). Evaluation used 30% of the 203 cases, rounded up to 61 cases, giving 305 tasks per type and 915 per model, with one 3,600-second session per task and network access open.
Why This Matters
Impact on research. The paper reframes benchmark contamination in smart contract security as a task-design problem rather than a network-restriction problem. Restricting network access, as some prior benchmarks do, also blocks legitimate agent behavior like installing dependencies or reading documentation; changing the protocol conditions instead removes the shortcut while leaving the agent's environment realistic. The finding that diagnoses and repairs often survive mutation while attack code does not is a useful distinction for anyone designing exploit-focused evaluations.
Real-world applications:
- Smart contract audit firms can use mutation-style checks to test whether a candidate auditor or tool genuinely understands a vulnerability or is pattern-matching a known incident.
- LLM agent and tooling developers get a benchmark whose answers cannot be retrieved verbatim, giving a cleaner signal on adaptation ability for security agents.
- Incident response and bug bounty operations can reuse the pipeline to turn a fresh on-chain incident into validated exploit and patch exercises quickly, rather than commissioning manual task development.
- Protocol teams and reviewers benefit from the paired vulnerable/repaired protocol construction and from the observation that attack-blocking scores alone overstate repair quality by 10.2–13.4 percentage points.
Industry relevance. With SlowMist recording 2,277 incidents and roughly $38.6 billion in losses across the blockchain ecosystem as of September 2026, automated evaluation of the agents proposed for blockchain security analysis is directly tied to how much those agents can be trusted. Lifelong construction matters commercially because security tooling goes stale quickly; a benchmark that ingests new cases through the same validated procedure avoids repeated manual rebuilds.
Future Directions
- Calibrating mutation difficulty. All 95% confidence intervals for success-rate changes from Base to Combined changes include zero, so the authors' own sample cannot establish that more mutations reliably make tasks harder. Whether mutation strength can be graded or targeted is open, especially since exploit success rose for four of six models.
- Extending beyond the current scope. The collection covers public EVM-oriented incidents and audit findings from DeFiHackLabs, BlockSec, and Web3Bugs, with reconstruction rejecting cases that need unavailable services or extra attacker permissions. Whether the approach transfers to other virtual machines, non-DeFi contract classes, or richer protocol state is untested.
- Making non-attack answers harder to reuse. Because 63 of 89 correct detect reports and 57 of 93 successful patches retained credit after mutation, the detect and patch tasks remain partly answerable from public material. Designing mutations that also perturb the diagnosis or repair requirement is a natural next question.
- Scaling and standardizing lifelong ingestion. The framework is designed to absorb new incidents through the same validated pipeline, but accepted-version yields (82.0% for the full method on 122 proposals) and construction costs ($1.50 per accepted version) determine how practical continuous, fully automated refresh is at larger scale.
Target Audience
Researchers and practitioners working on LLM-based security agents, smart contract auditing, and benchmark design will get the most from this paper. It is also relevant to blockchain security teams evaluating tooling, to benchmark maintainers concerned with contamination and data leakage, and to graduate students studying automated test and task generation via mutation. Readers need some grounding in smart contracts and agent evaluation to follow the construction details, but the central argument about copying versus adapting is accessible more broadly.
Authors’ abstract
AI agents can inspect code and run tests to support smart contract security. SlowMist reports about…
The full abstract on arXiv