Skip to content
Crypto.world

Research

Reentrancy Detection in the Age of LLMs

Reentrancy — where an attacker re-enters a contract during an external call and drives it into an unintended state — is described in the paper as the archetypal Ethereum vulnerability, with roughly…

Reentrancy Detection in the Age of LLMs
arXiv
2603.26497
Published
2026-03-27
Authors
Dalila Ressi, Alvise Spanò, Matteo Rizzo, Lorenzo Benetollo, Sabina Rossi

AI summary

Overview

Research area: Smart contract security — specifically, detecting reentrancy vulnerabilities in Ethereum contracts written in modern Solidity (0.8+), and benchmarking traditional static analyzers, machine-learning detectors, and large language models against each other.

Technical level: Intermediate. Readers should know roughly what a smart contract, an external call, and the checks-effects-interactions (CEI) pattern are; no formal-methods background is required.

Scope (one sentence): The paper builds two manually verified benchmarks (432 real-world contracts and 143 handcrafted stress-test contracts) and uses them to measure how reliable and robust 12 formal-methods tools, 10 ML models, and 9 LLMs are at reentrancy detection.

What This Paper Is About

Reentrancy — where an attacker re-enters a contract during an external call and drives it into an unintended state — is described in the paper as the archetypal Ethereum vulnerability, with roughly $908.6M in losses attributed to reentrancy-based attacks. The detectors built to catch it, however, have not kept pace with Solidity: many support only legacy compiler versions, crash on modern syntax, or encode definitions of reentrancy tied to the 2016 DAO exploit. The paper's goal is to quantify how dependable this tool ecosystem still is, and to test whether general-purpose LLMs offer a more robust alternative.

Key Contributions

  1. A dependability-driven assessment of reentrancy detectors. The authors frame evaluation around two properties — reliability (correctness of predictions, i.e., few false positives/negatives) and robustness (staying functional across evolving Solidity versions and contract patterns) — and argue this is the first systematic evaluation of the long-term dependability of reentrancy tools.

  2. Two complementary datasets. An Aggregated Benchmark of 432 contracts consolidated and relabeled from three prior "manually verified" datasets, and the Reentrancy Scenarios Dataset (RSD), 143 handcrafted minimal working examples (under 30 lines each) that isolate individual reentrancy patterns across a scenario taxonomy.

  3. A unified three-step labeling procedure. Rather than adopting any single tool's rule, the authors define reentrancy operationally via function unfolding, detection of external calls followed by side effects, and a semantic comparison between reentrant and flattened sequential execution.

  4. LLMs as both detectors and human-in-the-loop assistants. The paper shows several LLMs beating existing detectors zero-shot, and documents how LLM explanations helped reviewers spot mislabeled or ambiguous contracts. All code, datasets, and prompts are released in an anonymized repository.

Main Findings

  • LLMs lead on the real-world benchmark: On the Aggregated Benchmark, traditional tools and ML models reach up to 0.87 F1, while the best LLMs reach 0.96 in a zero-shot setting.
  • Stress tests break most tools: On the RSD, most tools fail on multiple scenarios; the top performer reaches an F1 of 0.76, whereas the strongest model attains 0.82.
  • The tool ecosystem is fragmented: Each analyzer encodes its own incompatible notion of reentrancy. Mythril and Conkas flag a low-level CALL forwarding more than 2,300 gas followed by an SSTORE; Slither expands the CEI rule into five distinct variants; Solhint reduces reentrancy to pure CEI violations; ConFuzzius encodes the DAO exploit explicitly; sFuzz only labels a contract vulnerable if a synthesized attacker contract actually reenters during fuzzing. Tools disagree on whether unlimited-gas high-level calls count, whether delegatecall is always reentrant, and whether mutexes or modifiers are adequate guards.
  • Prior "manually verified" labels are noisy: Of 145 potentially reentrant contracts in the aggregated pool, 28 were relabeled safe; of 291 sampled potentially safe contracts, 5 were relabeled reentrant. The final benchmark contains 432 contracts — 122 reentrant and 314 safe.
  • Legacy code dominates prior datasets: Most contracts in the source pool are written in Solidity v0.4.x or v0.5.x, with only a few in v0.6.x; some fail to compile because of a missing pragma or random code errors.
  • LLM explanations catch edge cases — but often hallucinate: When an LLM disagreed with the reviewers' label, the reviewers examined its explanation; in most cases the disagreement was a hallucination, but in a minority the model highlighted subtle behavior the humans had overlooked.
  • Maintenance decay undermines ground truth: Because most ML detectors are trained on datasets labeled by the same static tools, tool errors, disagreements, and crashes propagate directly into training labels.

Methodology in Plain English

The authors started by reading the source code and documentation of the main reentrancy analyzers to extract the exact rule each one implements — an exercise that revealed mutually incompatible definitions, and in some cases rules that were never actually implemented in the released tool.

They then built two datasets. The Aggregated Benchmark pools three prior datasets that explicitly claim human labeling — the Consolidated Ground Truth (CGT) dataset, HuangGai, and the reentrancy study by Zheng et al. (CGT is itself an aggregation of 13 datasets, including ZEUS, SolidiFI, SmartBugs Curated, and the SWC registry). After deduplication and filtering to compilable contracts, this yielded 73,579 unique contracts: 145 potentially reentrant and 73,434 potentially non-reentrant. For the safe group they sampled from contracts that at least 4 of 5 tools (Oyente, Mythril, Securify, Smartian, Sailfish) considered safe, using regular expressions to ensure inclusion of patterns known to confuse analyzers. Three authors with complementary expertise (programming languages/formal methods, Ethereum development, and AI-based contract analysis) inspected every contract in multiple rounds.

The RSD was built the other way around: instead of sampling reality, they wrote minimal contracts for each scenario family in a taxonomy (basic DAO, single/cross-function mutexes, modifier guards, send/transfer, ERC20 donations and staking, proxies, onlyOwner, loops, delegatecall, read-only reentrancy). Each scenario has variants created by controlled transformations — function folding, error raising, event emission, modifiers, inheritance, inlining, unchecked blocks, low-level static calls, and in-place subtraction.

Labeling follows a three-step procedure. Step 1 unfolds local function calls (but not this.f(...), which the compiler treats as external) and normalizes modifiers and inheritance. Step 2 searches the unfolded code for external calls followed by side effects; if none exist, the contract is safe. Step 3 is semantic: it compares the state produced by nested reentrant execution against the state produced by the same calls run as independent sequential transactions. If the two coincide, the contract is safe; otherwise it is vulnerable. The procedure deliberately flags delegatecall as exploitable and excludes emit and error from counting as effects.

Finally, four LLMs (gpt-4o, gpt-4.1, o3-mini, o4-mini) were queried for a label plus explanation on each contract; disagreements with the human labels were re-reviewed by the three authors. LLMs did not label the data — they served only as an additional signal.

Why This Matters

Impact on research: The paper challenges the practice of using tool-generated or loosely "manually verified" labels as ground truth. Because most ML-based detectors train on labels produced by the same static tools being evaluated, the field has been measuring detectors against inherited noise. The paper provides both a cleaner benchmark and an argument that benchmark construction must be tied to an explicit, execution-level definition of the vulnerability.

Real-world applications:

  • Smart contract audits: LLM-assisted triage could pre-screen contracts before a human auditor looks at them, with the caveat that no formal guarantee comes with it.
  • Developer CI pipelines: Knowing which analyzers crash or silently mislabel Solidity 0.8+ code tells teams which tools they cannot rely on as an automated gate.
  • ML training data curation: The relabeling results (28 of 145 reentrant contracts reclassified as safe) show how much label noise flows into models trained on legacy corpora.
  • Benchmark design for security tooling: The RSD's taxonomy of minimal working examples offers a template for stress-testing detectors on specific patterns rather than aggregate accuracy.

Industry relevance: The finding that leading LLMs outperform the majority of existing detectors — without version-specific maintenance — suggests a shift in how security tooling may be built and maintained, and raises concerns about the robustness and maintainability of the current reentrancy-analysis toolchain that audits and DeFi protocols depend on.

Future Directions

  • Hybrid approaches: The authors explicitly call for systems that combine the principled guarantees of formal tools with the adaptability of modern AI, rather than choosing one or the other.
  • Maintaining and versioning detectors: Given that many tools fail outright on modern syntax, an open question is how to keep analyzers current with Solidity releases — and how to measure that maintenance over time rather than at a single snapshot.
  • Better ground truth pipelines: The LLM-as-conflict-reviewer workflow worked well enough to catch overlooked edge cases; scaling human-in-the-loop, explanation-driven relabeling to larger corpora is an obvious next step.
  • Extending the stress-test methodology: The RSD covers reentrancy only; the same scenario-taxonomy approach could be applied to other vulnerability classes whose detectors rest on equally fragmented definitions.

Target Audience

Smart contract security researchers and auditors; tool builders maintaining static analyzers or ML-based vulnerability detectors; ML practitioners who train on smart-contract datasets and need to know how noisy the labels are; and blockchain engineers deciding what to put in a pre-deployment security pipeline. The paper is also a useful case study for anyone studying benchmark quality and label noise in AI-for-code more broadly.

Authors’ abstract

Reentrancy remains one of the most critical classes of vulnerabilities in Ethereum smart contracts, yet widely used detection tools and datasets continue to reflect outdated patterns and obsolete Solidity versions. This paper adopts a dependability-oriented perspective on reentrancy detection in Solidity 0.8+, assessing how reliably state-of-the-art static analyzers and AI-based techniques operate on modern code by putting them to the test on two fronts. We construct two…

The full abstract on arXiv
Read the original paper