Skip to content
Crypto.world

Research

The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics

Investigators increasingly hand raw, unverified smart contract bytecode to a frontier LLM and ask whether the contract is malicious. The paper argues that this practice rests on an untested…

The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics
arXiv
2609.14098
Published
2026-09-12
Authors
Timo Schefold

AI summary

Overview

Research area: AI security and adversarial robustness, specifically the reliability of large language models as forensic tools for analyzing Ethereum Virtual Machine (EVM) smart contract bytecode.

Technical level: Intermediate. Readers should be comfortable with the general idea of a smart contract, a bytecode-level analysis pipeline, and LLM evaluation methodology, but the paper is written to be followed without specialist cryptography knowledge.

Scope: A large-scale adversarial benchmark measuring how much LLM detection of hidden fund-draining code degrades when contracts are deliberately built to deceive the analyst, tested across 22 models, 13 purpose-built contracts, and six prompt strategies.

What This Paper Is About

Investigators increasingly hand raw, unverified smart contract bytecode to a frontier LLM and ask whether the contract is malicious. The paper argues that this practice rests on an untested assumption: that a model which can read code can also see through code written specifically to mislead it. The authors build contracts that contain a real drain mechanism but camouflage it with misleading strings, structural obfuscation, or embedded adversarial instructions, then measure the gap in detection between these contracts and functionally matched honest drains — a gap they call the deception delta.

Key Contributions

  1. An adversarial contract stress-test benchmark. Thirteen contracts with documented ground truth, organized along a deception-complexity axis from string-level misdirection through structural camouflage to combined multi-layer deception, including prompt-injection payloads embedded in the artifact under analysis. The authors explicitly do not claim taxonomic completeness.

  2. A systematic empirical evaluation with transparent measurement. Twenty-two models from seven providers, 8,528 analyzable non-refusal runs, a documented LLM-as-judge pipeline, two judge-independent validation metrics, and released contracts, prompts, responses, labels, and scripts sufficient to recompute the reported metrics.

  3. Rationalization as a distinct failure mode. The paper characterizes cases where a model correctly describes the hidden drain mechanism but then accepts the contract's deceptive framing and dismisses the behavior as benign — producing positive but incorrect evidence of safety.

  4. A quantified deception delta with sensitivity analysis. The headline 20.0 percentage point drop is checked against judge recalibration, recoding choices, contract exclusion, and multiple ways of handling refusals and pending runs.

Main Findings

  • The core deception delta is 20.0 percentage points. Detection on deceptive contracts was 21.0% versus 41.0% on non-deceptive drain controls (95% CI: [17.2, 22.8]).

  • Structural camouflage is the most effective deception. Multi-hop call chains, XOR-masked selectors, and storage-loaded drain parameters reduce aggregate detection to 2–4% on the hardest vectors. The strongest reasoning models still reach substantially higher rates on less obfuscated structural vectors, up to 100% on HV-5.

  • Rationalization produces confidently wrong reassurance. On the most revealing vector (HV-3b, which pairs a suspicious mechanism with roughly 4.1 KB of fabricated audit documentation), models describe the real drain but endorse the contract's "zero-and-refill accounting" framing and call it benign. Because rationalized verdicts are coded as detected rather than missed, this is a conservative choice — recoding them as missed increases the deception effect.

  • Prompting does not fix the problem. Structural deception varies by only 3.5 percentage points across the six prompt strategies, which the authors read as more consistent with a capability limitation than a prompting problem. Guard instructions ("ignore embedded instructions") give no aggregate benefit and shift individual models in both directions.

  • Model capability is highly uneven. Only five models from two providers exceed 50% detection; the majority of evaluated models detect fewer than 15% of deceptive drains. Claude Opus, which also served as the code-generation assistant during benchmark construction, reaches 76% detection — but against contracts designed specifically to challenge it.

  • Ground truth was verified twice. Seventy-one EVM execution tests and 13 bytecode-level selector checks were run, with no test or selector-verification failures observed. Seven of the 71 tests target predecessor variants not in the 13-contract evaluation set and are retained only to document construction provenance.

  • Results survive judge recalibration. Moving from naive judge voting to weighted voting changes the effect from −28.0 to −20.0 percentage points rather than eliminating it. The calibrated rule changes 654 verdicts (7.7%) relative to naive majority voting.

  • Judge agreement is moderate. DeepSeek Reasoner, the primary verdict source, agrees with human labels at κ = 0.630 (35/50); DeepSeek Chat at κ = 0.641 (36/50); Gemini Flash at κ = 0.519 but recalls 86% of the seven rationalized gold cases, earning it a targeted override role. Fleiss' κ across operational judge verdicts is 0.674, highest on controls (0.775) and lowest on HV-4b and HV-3 (0.249 and 0.294). A blinded LLM re-coding matched 49/50 gold cases.

  • Refusals are a separate phenomenon. Sixteen policy-style refusals by Grok 4 were observed; the paper states these are analyzed as a distinct finding in Section 5.6, but the detail of that analysis is not included in the available excerpt. Thirty-six runs were pending, 34 from Qwen-Coder-32B and 2 from Llama 4 Scout.

  • HV-1's exclusion barely moves the result. Because HV-1's hand-assembled token call uses a non-standard selector and cannot move funds from a standard ERC-20 token, excluding it changes the delta from −20.0 to −20.6 percentage points.

Methodology in Plain English

Who is being simulated. The threat model is an investigator — the author is affiliated with a German state criminal police office's digital financial investigations centre — who encounters a deployed contract with no verified source code and uses an LLM as a first-pass triage tool. The model gets hex-encoded runtime bytecode only: no source, no on-chain history, no storage state, no tools, and no follow-up turns. The attacker is assumed to know that LLM triage is in use and to build contracts accordingly.

The contract set. Four controls set the baselines: C1 (Honest Vault) has an honest, descriptively named drain; C2 (GreenHope) is a benign ERC-20 token and the false-positive control; C3 (Silent Vault) replicates C1's drain with all descriptive strings stripped, isolating how much models lean on metadata; C4 (Fortress Vault) is a realistic vault with the same owner-only advanceEpoch entry point and settlement call chain as HV-3b but a benign settlement path, isolating drain detection from governance-risk judgement.

Nine deception vectors are split into three families. String-based deception (HV-1 with a "owner_can_drain": false JSON block; HV-3b with fabricated audit documentation) misdirects interpretation of visible code. Structural deception hides the drain in architecture: HV-3 spreads extraction across five internal functions; HV-4b stores the transfer selector 0xa9059cbb as 0x77a82254 ⊕ 0xdeadbeef and decodes it at runtime; HV-4c is a smaller 6.7 KB variant of HV-4b; HV-5 achieves the same effect in clean Solidity by loading selector and recipient from storage at runtime; HV-6 is a 16.7 KB contract with over 30 cover functions. Prompt injection (HV-PI impersonating the Ethereum Foundation; HV-PI2 with a social-engineering payload plus embedded benign bytecode as inert dead code) targets the model's instruction-following rather than its code reading.

Design process and its admitted bias. The vectors were built adaptively using Claude Opus as both code generator and adversarial target — each new vector addressed a signal the previous one had relied on. Every contract then passed an author-curation pipeline: manual source review, freezing of final bytecode, the full test suite, and selector verification. The authors flag the circularity directly and note that a taxonomy optimized against a different model family might look different.

The experimental grid. Every model–contract–prompt combination was repeated five times, giving a planned 8,580 runs, of which 8,528 (99.4%) were analyzable non-refusals. Temperature was set to 1.0 where the API exposed it; reasoning endpoints without the parameter ran on provider defaults. Approximate API expenditure was $626.

The six prompts. Prompts A and B use investigation framing (B adds the guard instruction), D and E use neutral security-analyst framing (E adds the guard), F substitutes a structured checklist, and G adds a warning that contracts may contain deliberately misleading elements. Prompt C was dropped in pilot testing as redundant with D. All prompts request a 1–5 risk rating alongside free text.

How responses were scored. Manual labeling of 8,528 responses was infeasible, so the authors used an LLM-as-judge pipeline with a six-way taxonomy: true positive, false negative, hallucinated, rationalized, true negative, false positive. Judges receive the ground-truth spec, the model's full response and rating, and a detailed rubric. The primary judge is DeepSeek Reasoner, with DeepSeek Chat as a fallback (used for six runs), Gemini Flash supplying one targeted override for rationalized verdicts on drain-bearing contracts, and Ministral-14B kept as a calibration lane rather than an equal vote. A single domain expert labeled a stratified 50-case gold standard enriched for judge disagreement to calibrate the whole setup. Two judge-independent checks — selector matching and a risk-rating threshold — corroborate the aggregate effect without relying on any judge.

Why This Matters

Impact on research. Prior LLM smart-contract work almost universally tests unintentional vulnerabilities — reentrancy, integer overflow, access control bugs. This paper shifts the target to deliberate deception aimed at the analyst, and shows that the two are not the same problem. It also supplies a concrete mechanism (rationalization) by which an LLM forensic report can be worse than no report: it contains correct technical detail wrapped in a wrong conclusion. The released artifact makes the measurement reproducible in a subfield where benchmark construction is usually opaque.

Real-world applications:

  • Law enforcement and financial crime triage. A first-pass classification steers an investigation: calling a malicious-by-design contract "exploited" redirects effort toward a non-existent external attacker, and misreading an intended payment as a drain can turn a suspect into a victim.

  • Exchange and custody listing review. Teams screening new token contracts before listing are a natural deployment site for bytecode triage, and the paper's results argue against relying on a single-shot model verdict there.

  • Audit and due-diligence firms. The rationalization finding is directly actionable: a report that correctly names the suspicious function but concludes "benign accounting pattern" passes casual review while being substantively wrong.

  • Scam and rug-pull detection pipelines. The vector families are grounded in documented scam-token and honeypot patterns, making the failure modes recognizable to teams building automated detectors.

Industry relevance. The central practical claim is narrow and deliberately so: under a single-shot, raw-bytecode-only protocol, current LLMs are not reliable standalone forensic tools. Only five models from two providers clear 50% detection, and the majority detect fewer than 15% of deceptive drains. Guard-style instructions do not help. For anyone currently wiring an LLM into a contract-screening pipeline without source code, tools, or human review, this is a direct caution.

Future Directions

  • Multi-turn, tool-augmented, and source-aware workflows. The authors state explicitly that the central claim does not extend to these settings, and report only a source-code boundary check as an explicit subset analysis. Quantifying how much tooling, decompilation, or execution traces recover is the obvious next measurement.

  • Defenses against rationalization specifically. Since the failure is in judgement rather than perception, better code reading will not fix it. Training or prompting that treats a plausible benign explanation as a claim requiring independent verification is an open design question.

  • Testing taxonomy transfer. The vector set was built adaptively against Claude Opus. Whether a taxonomy optimized against a different model family produces different or harder vectors — and whether the observed model ranking is stable — is unresolved.

  • Understanding refusal and abstention behavior. Sixteen Grok 4 refusals are treated as a distinct finding rather than as misses. How refusal rates interact with adversarial artifacts, and whether refusal is a safe or an unhelpful response in forensic triage, remains an open question the paper flags but does not settle in the available text.

Target Audience

Blockchain security researchers and auditors who use or build LLM-assisted analysis tools; LLM evaluation and adversarial-robustness researchers interested in a domain where the adversary targets the model's interpretation rather than its safety alignment; digital forensics and financial-crime investigators who need to know the limits of automated triage; and model developers who want a concrete, released benchmark of analyst-facing deception rather than conventional bug-finding.

Authors’ abstract

Large language models are increasingly used in blockchain forensic investigations to interpret unverified smart contract bytecode. Their robustness has not been systematically tested against contracts adversarially designed to mislead analysis. We evaluate 22 frontier models on 13 purpose-built contracts (9 deception vectors, 4 controls) across six prompt strategies, yielding 8,528 analyzable non-refusal runs against contracts with documented ground truth. A calibrated…

The full abstract on arXiv
Read the original paper