Research
PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts
Smart contract auditors must show that a reported vulnerability is real, and the standard way to do that is a proof-of-concept exploit: a runnable test that triggers the bug and asserts the resulting…

- arXiv
- 2511.02780
- Published
- 2025-11-04
- Authors
- Vivi Andersson, Sofia Bobadilla, Harald Hobbelhagen, Martin Monperrus
AI summary
Overview
Research area: Smart contract security, specifically automated generation of proof-of-concept (PoC) exploits, at the intersection of blockchain security and agentic AI for code.
Technical level: Intermediate. Readers should know roughly what a smart contract and a Solidity test are, but the paper's core loop (LLM + tools + feedback) is explained plainly.
Scope: The paper presents PoCo, an autonomous LLM-agent framework that turns an auditor-written natural-language vulnerability description plus a Solidity codebase into an executable Foundry test that demonstrates the vulnerability, and evaluates it on 23 real-world audit findings with three frontier models.
What This Paper Is About
Smart contract auditors must show that a reported vulnerability is real, and the standard way to do that is a proof-of-concept exploit: a runnable test that triggers the bug and asserts the resulting damage. Writing these PoCs by hand is slow, error-prone, and squeezed by tight audit schedules. The paper's goal is to automate that step, generating working Foundry PoC tests directly from the auditor's written vulnerability description, without the agent needing to discover the vulnerability itself.
Key Contributions
-
PoCo, an agentic blueprint for PoC generation. A single-LLM agent that works in a Reason–Act–Observe (ReAct-style) loop over file-system tools, a planning (
Todo) tool, and two smart-contract tools (smart-contract-compileandsmart-contract-test) built on Foundry, producing executable PoC tests. -
A patch-based correctness oracle. A new evaluation methodology that uses the developer-provided mitigation patch as an independent ground truth: a PoC counts as logically correct if at least one of its assertions fails when run against the patched code.
-
An experimental campaign over 23 real-world vulnerabilities and 3 models (Claude Sonnet 4.5, OpenAI o3, GLM 4.6), reported as generating three times more logically correct PoCs than the workflow baselines. Results are released at
github.com/ASSERT-KTH/PoCo-public/. -
The Proof-of-Patch dataset. An open, reproducible dataset linking real audit findings to their source code and mitigation patches (
github.com/ASSERT-KTH/proof-of-patch), described by the authors as the first smart contract dataset to link valid vulnerability findings with the security patches Web3 developers actually implemented.
Main Findings
-
Well-formedness (RQ1): Across the full campaign the paper reports that PoCo generates executable PoCs in 50 runs on the 23-item dataset with three models. Per-model and per-project breakdowns live in Table 2, which is not visible in the provided excerpt.
-
Logical correctness (RQ2): PoCo generates 32 correct PoC exploits for the real-world vulnerabilities, where correctness is defined by the PoC being blocked by the ground-truth mitigation patch.
-
PoCo beats both baselines by a wide margin: The authors state PoCo "consistently outperforms the Zero-shot and Workflow baselines" on both well-formedness and logical correctness, and that it produces "three times more logically correct PoCs than workflow baselines." The exact baseline counts are in the paper's result tables and are not reported in the excerpt available here.
-
Failure modes are categorized, not just counted. Failed runs are attributed to compilation failure (CF), no assertion (NA), ill-formed assertion (IA), max cost exceeded (MC, threshold $3), or max tool calls exceeded (MT, threshold 10), with a precedence order of MC > MT > CF > IA > NA so that resource-limit terminations are not miscounted as semantic failures.
-
Annotation detail matters (RQ3): The authors construct three cumulative annotation levels — high-level, detailed (adds code snippets and root-cause explanation), and procedural (adds a step-by-step exploitation scenario) — and test only Claude Sonnet 4.5 on the nine samples for which all three versions could be built. The direction of the effect is not reported in the available excerpt.
-
Patch-based validation is designed to tolerate incomplete patches. Because a PoC is marked correct if any assertion fails on the patched code, the oracle remains sound even if the developer's fix addresses only one variant of the vulnerability.
-
Cost is bounded and reported. The whole experimental campaign cost $135 USD, run on Ubuntu 22.04, a 36-core Intel Core i9-10980XE at 3.00 GHz with 125 GB RAM, at temperature 0 with seed 1615315.
Methodology in Plain English
The researchers gave an LLM agent a job and then let it work like a developer would.
-
Inputs. For each case, the agent gets three file paths: the vulnerable contract, the auditor's natural-language vulnerability description, and where to write the PoC test file.
-
Tools, not instructions. The agent can search the codebase (
glob,grep), read files, edit or write files, keep a to-do list of subtasks, and — crucially — compile the contracts and run the Foundry test. It decides on its own which tools to call, in what order, and with which flags; the authors deliberately keep the scaffold minimal, citing the "bitter lesson" principle. -
The loop. In a ReAct-style cycle, the agent reasons about the vulnerability, acts by writing or editing a test, observes compiler and test output, and revises. It is finished when the test compiles and its assertions pass on the vulnerable code.
-
Guardrails. Only an allowlisted set of CLI commands is available (none can send on-chain transactions), and all execution happens inside Docker containers so the agent cannot touch host secrets like SSH keys.
-
Checking correctness, not just compilation. A passing test only proves the code compiles and runs. To prove the test actually exercises the bug, the researchers rerun it against the officially patched version of the contract. If the PoC still passes on patched code, it never touched the real vulnerability. If at least one assertion fails on the patched code, the PoC is deemed logically correct.
-
Comparing against simpler approaches. Because prior tools (REX, A1, Quimera) either were not publicly available or could not be run, the authors built two controlled baselines instead: Zero-shot (one shot, no feedback) and Workflow (a fixed two-phase pipeline with compile/test feedback but no autonomous planning or codebase search). All three approaches received the same inputs and used the same stopping criteria.
-
Finding the data. They pulled 3,814 audit findings from the Solodit aggregator across the ten OWASP Smart Contract Top 10 categories, ranked them with an LLM-as-Judge quality rubric (bad / fair / good / excellent), manually inspected the top 100, and kept the 23 whose projects were publicly available, Foundry-testable within a one-hour setup limit, judged valid, and backed by a real patch.
Why This Matters
Impact on research. The paper contributes a reusable evaluation idea — treating developer patches as an independent oracle for exploit validity — that sidesteps the usual problem of judging generated exploits by hand or by mere compilation. Combined with the open Proof-of-Patch dataset, it gives the field a reproducible benchmark built entirely from real audit findings rather than synthetic or ERC-standard toy contracts.
Real-world applications:
- Audit reports: auditors can attach an executable PoC to every finding, strengthening severity claims and, the authors note, improving the likelihood of a higher reward on competitive audit platforms like Code4rena and Immunefi.
- Developer remediation: the PoC doubles as a failing regression test that the development team can run before and after the fix.
- Pre-deployment gating: faster PoC turnaround lets teams catch and block exploitable code before it reaches an immutable on-chain deployment.
- Security tooling integration: PoCs are emitted as standard Foundry tests, so they drop into existing CI pipelines and fuzzing or analysis tools.
Industry relevance. With roughly $15 billion in on-chain losses as of October 2025, and audits constrained by limited budgets and tight timelines, cutting the manual cost of PoC writing targets a concrete bottleneck in the audit workflow rather than a hypothetical one.
Future Directions
-
Does annotation detail change success? RQ3 is run on only nine samples with a single model (Claude Sonnet 4.5), so how robust PoCo is to tersely written findings — the common case in practice — remains an open question, as does whether the finding generalizes across models.
-
Comparison against purpose-built exploit generators. The authors could not run REX, A1, or Quimera (implementations unavailable, or too immature to run), so how PoCo compares with specialized smart-contract exploit tools is still unmeasured.
-
Reducing cost and tool-call limits. With hard caps of $3 per run and 10 smart-contract tool calls, it is unclear how much performance is left on the table by these budget constraints, or how they should be tuned for harder vulnerabilities.
-
Beyond a single agent. The design deliberately uses one LLM agent with minimal scaffolding; whether multi-agent decomposition, richer Solidity-aware tools, or learning from the recorded trajectories (the paper logs full interaction traces and per-tool execution times) would push correctness higher is untested.
Target Audience
Smart contract auditors and security engineers who write vulnerability reports; blockchain security researchers building automated auditing and exploit-generation tools; LLM-agent researchers looking for a demanding, tool-heavy, real-world code domain; and dataset/benchmark builders interested in patch-linked evaluation methodology. Developers maintaining DeFi protocols will also find the Proof-of-Patch dataset useful as a corpus of real vulnerabilities paired with their fixes.
Authors’ abstract
Smart contracts operate in a highly adversarial environment, where vulnerabilities can lead to substantial financial losses. Thus, smart contracts are subject to security audits. In auditing, proof-of-concept (PoC) exploits play a critical role by demonstrating to the stakeholders that the reported vulnerabilities are genuine, reproducible, and actionable. However, manually creating PoCs is time-consuming, error-prone, and often constrained by tight audit schedules. We…
The full abstract on arXiv