# Beyond Detection: Agentic Attack Synthesis and Simulation for Smart Contracts

Crypto.world

> Current smart contract security tools — static analyzers, audit reports, and newer LLM-based auditors — mostly identify suspicious code and then stop. They rarely demonstrate whether a reported flaw…

**arXiv**: 2607.15673

**Published**: 2026-07-17

**Authors**: Xianhao Zhang, Jing Sun, Zijian Zhang, Ye Liu, Zhe Hou, Jiaqi Gao, Yuqiang Sun

## AI summary

## Overview

**Research area:** Smart contract security, specifically automated exploit generation (AEG) for Ethereum contracts using LLM-based multi-agent systems.

**Technical level:** Advanced. The paper combines formal notation (Bayesian inference framing, CEGIS-style search analysis, constraint schemas) with a full multi-agent engineering system and an empirical benchmark study.

**Scope (one sentence):** The paper proposes and evaluates KASS (Knowledge-Augmented Attack Synthesis and Simulation), a three-agent framework that turns smart contract vulnerability reports into executable, validated Foundry proof-of-concept exploits rather than stopping at detection.

## What This Paper Is About

Current smart contract security tools — static analyzers, audit reports, and newer LLM-based auditors — mostly identify suspicious code and then stop. They rarely demonstrate whether a reported flaw can actually be triggered, what transaction sequence is required, or how much money an attacker could take. The authors cite evidence that among 4,364 contracts reported as vulnerable, roughly 75% were considered unexploitable, meaning many findings are false positives or theoretical issues.

The paper's goal is to close this "detection-to-execution gap" by building a system that automatically plans an attack, writes it as a runnable Foundry test, executes it, and iteratively repairs or re-plans until the exploit either succeeds or is shown to be infeasible — producing concrete evidence of exploitability and quantified damage.

## Key Contributions

1. **The KASS framework.** A multi-agent system that decomposes smart contract automated exploit generation into three stages — planning, code generation, and execution-based testing — coordinated through explicit intermediate artifacts (a structured Attack Plan, a Solidity test contract, and a `Log.md` execution record) rather than one end-to-end prompt.
2. **Retrieval-augmented attack planning.** A Planner Agent that extracts a local dependency context around the vulnerability site via call-graph traversal, retrieves real-world audit findings from the Solodit knowledge base by vulnerability type, and synthesizes a five-field structured attack plan (Explanation, Preparation, Interaction, Post-State, Objective), with the Objective restricted to four explicit categories so plans are testable.
3. **Formal generation and validation constraints.** Three syntactic constraints embedded in the PoC-Generator Agent's prompt — version compatibility (`C_ver`), minimalism (`C_min`), and oracle embedding (`C_oracle`) — that bind the abstract plan to a compilable Foundry test whose final `require` statement encodes the success condition.
4. **Hierarchical dual-loop refinement.** A Foundry-Tester Agent with an inner loop that repairs code-level errors from execution traces, and an outer loop that escalates persistent failures back to the Planner for strategy-level replanning, analyzed through a CEGIS and information-gain lens.

## Main Findings

- **Overall success on SmartBugs-Curated:** KASS with GPT-5.1 generated executable exploits for **98 of 104 contracts (94.23%)**, versus 21 of 104 (20.19%) for the reproduced Claude Code baseline using the same DeepSeek-V3.2 backbone and the same success criterion — a gap of **74.04 percentage points**.
- **Comparison to reported baselines:** The reported REX result on the same 104-contract benchmark is **52 (50.00%)**, making KASS's 94.23% higher by **44.23 percentage points**. On the reentrancy subset, KASS w/ GPT-5.1 reaches **93.55% (29/31)**, compared with the reported **80%** AdvSCanner figure. The authors explicitly flag these as contextual references, not a controlled head-to-head comparison, because implementations are unavailable and success criteria, backbones, prompts, budgets, and environments may differ.
- **Backbone sensitivity:** KASS w/ Gemini-3-Flash and w/ DeepSeek-V3.2 both reach **88/104 (84.62%)**. Per-category results differ: Gemini-3-Flash gets 80.65% reentrancy, 86.67% arithmetic, 50.00% DoS, and 90.38% unchecked calls; DeepSeek-V3.2 gets 74.19%, 100.00%, 83.33%, and 86.54% respectively.
- **Per-category results for KASS w/ GPT-5.1:** Reentrancy 93.55% (29/31), Arithmetic 100.00% (15/15), DoS 83.33% (5/6), Unchecked Low Level Calls 94.23% (49/52).
- **Pre-trained knowledge alone is weak:** In the pure prompting baseline, GPT-5.1 achieves **33.65% (35/104)**, DeepSeek-V3.2 **9.62% (10/104)**, and Gemini-3 **4.81% (5/104)**. GPT-5.1's performance is uneven — 73.33% on arithmetic but only 12.90% on reentrancy.
- **Safety refusals materially suppress the pure-prompting baseline:** GPT-5.1 refuses **50.96%** of cases, concentrated in reentrancy (**23 of 31**) rather than arithmetic (**1 of 15**). Excluding refusals, its success rate rises to **68.63%**.
- **Real-world CVE validation:** On **11** CVE-tagged contracts, KASS successfully validates **9 cases**. Detailed per-case information is reported in the paper's Table V, which falls outside the supplied excerpt.
- **Cost profile is dominated by testing:** The Planner Agent averages **16,370 tokens and 95.1 seconds**; the PoC-Generator averages **20,241 tokens and 204.5 seconds**; the Foundry-Tester averages **50,669 tokens and 724.7 seconds**, with a maximum of **110k tokens and 1,867.4 seconds** on complex cases.
- **Token–time correlation:** Pearson correlation is moderate for the Planner (**r = 0.486, p < 0.001**) and strong for the PoC-Generator (**r = 0.917, p < 0.001**) and Foundry-Tester (**r = 0.877, p < 0.001**). The weaker Planner correlation is attributed to external retrieval latency.
- **Qualitative distinction from Claude Code:** The authors attribute Claude Code's failures to shallow test harnesses that deploy contracts or print logs without executing the vulnerable path or asserting attack success, and note that KASS's planning stage plus exploit-specific postconditions suppress this failure mode.
- **Not reported in the supplied excerpt:** The RQ3 ablation and LLM-backbone sensitivity results, the RQ5 quantification results beyond the abstract's claims about damage quantification and semantic false-positive filtering, and the detail of the representative WALLET contract failure case, which is cut off mid-sentence.

## Methodology in Plain English

The system is built as a team of three specialised agents, each with a narrow job, passing structured files to one another.

**Planner Agent.** It first trims the vulnerable contract down to only the code that actually matters — starting from the reported vulnerability location and following the call graph to collect state variables, functions, modifiers, and access checks that touch it. It then searches the Solodit audit database for real-world findings of the same vulnerability type and uses those as worked examples. From this it writes a JSON attack plan with five fields: a root-cause explanation, preparation steps (e.g., obtaining a flash loan), an ordered list of function calls, a measurable post-condition, and an objective drawn from four fixed categories (primary financial gain, strategic financial positioning, disruption/DoS, or manipulation of system behaviour). The authors justify this retrieval step with a Bayesian argument: without examples, the model's prior over "how reentrancy actually gets exploited" is broad and multi-modal; retrieved real reports act as evidence that sharpens it.

**PoC-Generator Agent.** It translates the plan into Solidity: preparation becomes the `setUp()` function (deploying the target, using Foundry cheatcodes like `deal` and `prank`), the interaction sequence becomes `testExploit()`, and the post-condition becomes a final `require` that reverts unless the attack succeeded. Three constraints are enforced through the prompt: the test's compiler version must match the target's, imports must be minimal (essentially just the vulnerable contract and Foundry's standard library), and the oracle must be present. It also emits an `attack.md` document explaining how the plan maps to the code.

**Foundry-Tester Agent.** It runs `forge test -vvvv`, reads the full trace (call stack, state changes, revert messages, gas), classifies the failure as a compilation error, runtime revert, or assertion failure, and edits only the attack contract — never the vulnerable one. Failures and their fixes accumulate in a history buffer that is fed back into the prompt. If the inner loop exhausts its budget without success, the outer loop sends the problem back to the Planner for a revised plan. Success requires three things at once: the syntactic constraints still hold, the target contract is unmodified, and the test passes — followed by one confirmation re-run.

**Evaluation.** They test on 104 SmartBugs-Curated contracts across four categories (Reentrancy 31 samples averaging 44 lines, Arithmetic 15 averaging 95, Unchecked Low Level Calls 52 averaging 130, DoS 6 averaging 49), plus 11 manually curated CVE-tagged contracts. Baselines are REX and AdvSCanner (numbers taken from their original papers, since neither is open-sourced), a reproduced Claude Code baseline run on the same 104 contracts under the same success criterion, and a pure-prompting baseline. Three LLM backbones are used: GPT-5.1, Gemini-3-Flash, and DeepSeek-V3.2. Three authors independently inspected each PoC and log. Experiments ran on Ubuntu 22.04.5 LTS with an i5-13400 CPU and 64GB of memory, built on Claude Code's sub-agent architecture with Solodit accessed via the search-solodit-mcp tool.

## Why This Matters

**Impact on research.** The paper reframes smart contract security from classification to verification: a vulnerability report is only useful if you can show the attack executes and what it costs. Its dual-loop design and structured-plan intermediate also give a reusable template for applying LLM agents to other domains where generated code must be proven to work against a real target, not just compile.

**Real-world applications:**

- **Audit triage:** Using generated exploit plans as a semantic false-positive filter in front of static analyzers, so auditors spend time on findings that are actually reachable.
- **Pre-deployment gating:** Running KASS against a protocol's contracts before launch to check whether flagged issues are genuinely exploitable and, if so, how much value is at risk.
- **Damage quantification for incidents:** Producing reproducible tests that measure attacker balance changes before and after, useful for post-mortems and for justifying bounty or disclosure decisions.
- **Bug bounty and disclosure workflows:** Giving reporters an executable PoC and a documented attack path instead of a textual claim.

**Industry relevance.** The motivation numbers are large: DeFi protocols managed more than US$69 billion in total value locked in 2026, while exploit losses reached US$512 million in 2025. With roughly 75% of reported vulnerabilities in one study judged unexploitable, tooling that separates real exposure from noise has direct economic value for protocols, auditors, and insurers.

## Future Directions

- **Extending beyond the four benchmark categories.** The evaluation covers reentrancy, arithmetic, DoS, and unchecked low-level calls. Whether the retrieval-plus-planning approach handles machine-unauditable classes such as price oracle manipulation — which require domain-specific knowledge rather than code-local reasoning — is not established by the reported results.
- **Reducing the testing agent's cost.** The Foundry-Tester Agent consumes the majority of tokens (averaging 50,669, peaking at 110k) and time (averaging 724.7 seconds, peaking at 1,867.4 seconds). Cheaper trace analysis or better-directed inner-loop repairs would make batch auditing practical.
- **Scaling real-world validation.** Only 11 CVE-tagged contracts are used, of which 9 validate. A larger and more diverse CVE corpus would test whether the 94.23% benchmark rate carries over to production complexity, multi-contract dependencies, and upgradeable proxies.
- **Characterising the failure boundary.** The paper begins a WALLET contract failure analysis to define KASS's capability limits, but the supplied excerpt cuts off before the findings. A systematic taxonomy of when the outer-loop replanning cannot rescue a case would tell users when to trust a "not exploitable" verdict.

## Target Audience

Security researchers working on automated exploit generation and LLM-agent systems; smart contract auditors and security engineers who want executable evidence rather than static warnings; tooling engineers building on Foundry and audit-knowledge bases like Solodit; and graduate students studying the application of multi-agent LLM workflows to program analysis and verification. Readers should be comfortable with Solidity, Foundry testing conventions, and the basics of how LLM agents are prompted and constrained.

## Authors’ abstract

Smart contract vulnerabilities pose severe financial risks, yet existing security tools largely stop at vulnerability detection, offering limited support for explaining whether reported flaws are exploitable, how attacks unfold, and what concrete damage they cause. To bridge this gap, we propose KASS (Knowledge-Augmented Attack Synthesis and Simulation), a multi-agent framework for executable smart contract exploit verification. KASS decomposes automated exploit generation…

[The full abstract on arXiv](https://arxiv.org/abs/2607.15673)

[Read the original paper](https://arxiv.org/abs/2607.15673)

---

Canonical: https://crypto.world/paper/beyond-detection-agentic-attack-synthesis-and-simulation-for-smart-2607.15673
