# ABF-Bench: A Verifiable Benchmark for Agentic Blockchain Forensics

Crypto.world

> Turns fund tracing into a ledger-cited, automatically gradable task built from real laundering cases, giving forensics-agent builders a shared yardstick instead of bespoke demos.

**arXiv**: 2610.23656

**Published**: 2026-10-16

**Authors**: Xiaohang Yu, Hejia Geng, William Knottenbelt

## AI summary

## Overview

**Research area:** Blockchain forensics and anti-money-laundering, evaluated through LLM-agent benchmarks (cryptography and security; arXiv 2610.23656v1, cs.CR).

**Technical level:** Intermediate. The paper is readable with some familiarity with blockchain tracing concepts (transfers, UTXOs, bridges, mixers, custody) and with how LLM agents and verifiable benchmarks work; the formal definitions are contained in a small number of sections.

**Scope:** ABF-Bench is an interactive, budget-metered benchmark that turns real laundering incidents into verifiable fund-tracing tasks, where every claimed fund movement must cite ledger records the solver actually retrieved, and an accompanying agent harness is compared against general-purpose agents and fixed-order tracers.

## What This Paper Is About

Fund tracing on public blockchains asks where stolen value came from and where it went, but launderers split, swap, bridge, and park funds at mixers and custodial services, and they change tactics between incidents, so fixed procedures fall behind. The authors argue that because public ledgers immutably record every movement, most of an investigation is independently checkable, which makes forensics a good fit for verifiable agent tasks. Their goal is to build such a benchmark, with tasks derived from real laundering activity, and to measure how well LLM agents, purpose-built harnesses, and deterministic tracers perform under a shared query budget.

## Key Contributions

1. **Task formulation.** The authors formulate evidence-backed blockchain fund tracing as an interactive, verifiable agent task: a task is a tuple of admitted ledger records, seed fund states, a declaration (scope and accounting convention), and admitted operation types, and every submitted claim must cite ledger locators that a deterministic verifier checks.
2. **ABF-Bench and its case-to-task pipeline.** They build the benchmark with a reusable generator that turns a real laundering incident into well-posed cross-chain tasks, using conservative rules that halt tracing wherever records leave a continuation unresolved, so each reference graph is an under-approximation of the flow, and each reference edge carries a witness retrievable within the task's query budget.
3. **The ABF agent harness.** They propose a harness built around a working memory (findings, leads, notes), supporting tools (memory operations, a submission preview, context compaction), and five prompt-level heuristics, and they evaluate it alongside general-purpose agents and fixed-order tracers on the same evidence interface.
4. **Empirical comparison of solver families.** They report edge recall, fidelity, edge precision, budget used, output tokens, and API cost across the harness, Codex, Claude Code, and breadth-first, depth-first, and greedy tracers, and diagnose where each family gains and loses across four investigation stages (search, recognition, evidence construction, reporting).

## Main Findings

- **Context and motivation:** attackers stole $2.2 billion from cryptocurrency platforms in 2024 alone, and learned detectors degrade sharply after a single market shutdown, which motivates case-adaptive investigation.
- **Harness beats general-purpose agents on recall:** the ABF harness reaches **0.75** edge recall at **0.98** fidelity and **0.92** edge precision, against **0.47** recall for Codex with the same model (**gpt-5.5** at high reasoning effort) and **0.10** for Claude Code (**claude-opus-4-8** at high effort). Because the harness and Codex share a model, the 0.75-versus-0.47 gap is attributed to harness design.
- **Claude Code's low score is a retrieval problem:** its 0.10 is set by the retrieval condition (a reference edge counts only if the run retrieved every record it cites); scored on verdicts alone it reaches **0.54**, because it cites records it never retrieved.
- **Fixed-order tracers still win on final recall:** breadth-first tracing reaches **0.88** recall (dfs **0.77**, greedy **0.54**), and the harness beats bfs on only **7 of 31** tasks. Excluding tracer runs that ended without a submission raises bfs only to **0.91**, so the ranking holds.
- **Agents spend less of the budget:** the harness finishes with **47%** of its budget spent (Codex **33%**, Claude Code **16%**), while the tracers spend **0.87–0.92**. The tracers' higher edge precision loss is explained by service sweeps past undisclosed custody that produce valid but out-of-scope edges, giving tracers edge precision of **0.63–0.78**.
- **Faster early evidence acquisition (H1):** at a quarter of the budget the harness reaches **39%** evidence recall against **35%** for the best-in-hindsight tracer; at half the budget it reaches **70%** against **64%**. At full budget it reaches **75%** against **91%** for bfs. The best-in-hindsight tracer overtakes the harness after **54%** of the budget, and the harness leaves **53%** of its budget unused on average. The harness leads at its own stopping point on **26 of 31** tasks.
- **Agents stop early:** transcripts show Codex declining further outflows as too costly with two thirds of its budget left, and Claude Code declaring completion after **17 of 428** actions. Unexamined branches hold a quarter of all reference edges for the harness and more than half for Codex.
- **Recognition of undisclosed custody (H2):** the harness recognizes undisclosed custodial services the tracers walk past, at **89% vs. 0%**. Each agent stops at three quarters or more of the reachable undisclosed-custody points it listed, while every tracer stop came with its budget spent. bfs instead emits **367** valid edges beyond undisclosed custody, and reaches **87%** of reference edges into these points against **63%** for the harness.
- **Operation claiming:** the harness claims **83%** of retrieved operations against **85%** for bfs; Codex claims only **44%** of retrieved crossings, failing to carry the deposit memo to the Bitcoin payout. Every evaluated solver has at least **94%** of its operations accepted by the verifier.
- **Reliability (H3):** the harness and Codex match bfs fidelity (**0.98**), while Claude Code falls below at **0.95**. Unreported transfers are **2.5%** for the harness, **1.0%** for Codex, **1.8%** for Claude Code, **0.7%** for bfs, **1.1%** for dfs, and **0.0%** for greedy.
- **Recall degrades with branching more than size:** from the smallest to the largest third of tasks by reference edges, harness recall drops from **0.82** to **0.69**, and both general-purpose agents lose a larger share. Harness recall correlates more strongly with the largest single-holding fan-out (**Spearman rho = -0.66**) than with the number of reference edges (**-0.41**).
- **Run-to-run noise:** within-task standard deviation of recall across repeated runs is **0.18** for the harness and **0.17** for Codex, while mean recall agrees within **0.01**.
- **Possible case knowledge in models:** though the served text omits both "Bybit" and "Lazarus", Claude Code names the incident in **21 of 31** surveyed transcripts, against **2 of 33** for Codex and **0 of 33** for the harness; the authors note a mention only shows the knowledge is available.
- **Benchmark composition:** the case is the Bybit theft of February 2025, followed for **90 days** across Ethereum and Bitcoin, seeded from the FBI notice and Bybit's published incident addresses. Of **58** tasks produced by the authoring pass, **33** pass the release filter and are published; **31** are evaluated (**2044** reference edges total, **20–107** per task, budgets **98–510**). One task was excluded after repeated environment failures for one arm, and one after its amended reference graph outgrew its published budget.
- **Task features:** **1658** transfer edges, **166** exchange edges through venues, **220** cross-chain crossing edges, **503** Bitcoin (UTXO) edges; **113** edges into undisclosed custody versus **41** into disclosed custody; tasks disclose **20** custodians; the largest holding fan-out is **28**.
- **Not reported:** the truncated content does not report a completed H3 conclusion beyond the fidelity and unreported-transfer figures above, nor a measurement of what continuing past the agents' stopping points would recover ("remains unmeasured").

## Methodology in Plain English

The authors treat an investigation as a game with a meter. A solver receives a public brief: seed addresses, block windows, a hop cap, which bridges are followed, disclosed stops, available query actions, submission rules, and a fixed budget. It then queries ledger data through a shared interface, where each well-formed query costs **1** and each page of an activity listing counts as one query. When it submits, it files grouped operations, each citing the exact ledger records that support it.

Scoring is deliberately split in two. A deterministic verifier judges each submitted operation only against the records it cites, returning valid, invalid, or indeterminate, and it accepts any evidence that satisfies one of the stated operation rules, so an alternative to the reference witness still passes. A separate grader then compares the verdicts to a hidden reference graph and computes edge recall (the primary score), fidelity, and edge precision. A reference edge counts as recovered only if the report contains it in a valid operation *and* the run itself retrieved every record it cites, which enforces "cite only what you were served."

The reference graphs are built by a generator with two nested loops: a within-ledger loop that expands the frontier and replays records to decide which holdings contain traced value (with purity as the delimiting rule, and non-dust inputs on Bitcoin), and an across-ledger loop that links evidenced deposits with payouts and feeds return deliveries back into Ethereum, repeating until successive rounds coincide. Tracing halts conservatively wherever records leave a continuation unresolved, including at custody, unsupported bridge continuations, and mixer deposits. Human reviewers audit the graph, test the rules through two independent readings (separately implemented fixed-order tracers and the agents themselves), and each task is published only if the verifier accepts every retained edge's witness. Budgets are calibrated from the reference plan alone: **B(t) = ceil(2(η(t) + δ(t)))**, where η(t) counts the reference solver's actions and δ(t) one activity listing per distinct source holding.

Solvers compared under this setup are three fixed-order tracers implemented independently of the generator and verifier (breadth-first, depth-first, and greedy by largest incoming amount, run under a 2 GB memory limit), two command-line general-purpose agents restricted to the benchmark tools, and the ABF harness using the same model as Codex. Failed runs stay in the suite with zero recall, and finished-run means are also reported.

## Why This Matters

**Impact on research.** The paper adds a domain where difficulty requires no expert curation—real adversaries generate the instances and the public ledger keeps every claim checkable—and where the same verdicts that score agents could serve as reinforcement-learning signals. It also separates evidence *acquisition* from evidence *reporting* in scoring, which is unusual among agent benchmarks and exposes failure modes (overconfident citation, premature stopping) that aggregate recall alone would hide.

**Real-world applications:**

- Anti-money-laundering compliance and transaction-monitoring teams that need auditable, citation-backed traces rather than opaque risk scores.
- Incident response after exchange or bridge exploits, where investigators must show which records support each claimed hop.
- Cross-chain and bridge analytics, where deposit-and-payout pairing across ledgers is the crux of proving a crossing.
- Exchange and custodian onboarding/risk review, where recognizing an undisclosed custodial service from behavior alone changes whether a trace continues.

**Industry relevance.** The measured gap between the harness and Codex on the same model (0.75 vs 0.47) shows that agent engineering, not just model choice, drives forensic performance, and the reported API costs ($9.4 for the harness, $2.7 for Codex, $6.5 for Claude Code per task on average) give practitioners a concrete cost signal. The finding that agents leave roughly half their budget unspent points directly at a deployable improvement.

## Future Directions

- **Closing the early-stopping gap.** Every agent stops early and bfs retains the best final recall (0.88), which the authors present as headroom; agents that exhaust their leads rather than declaring completion are the obvious next step.
- **Handling branching rather than size.** Harness loss tracks the largest single-holding fan-out (rho = -0.66) more than edge count, suggesting search policies that allocate budget across wide fan-outs need design work.
- **Tracing past undisclosed custody.** Agents recognize custody but then stop, while tracers continue and reach more reference edges into those points (87% vs 63%); combining recognition with continuation is left open.
- **Extending to new ledgers and evidence policies.** The authors note that a new ledger or operation type requires new evidence rules and implementation, and that admitting other record classes such as protocol attestations or published internal records would define evidence policies of possibly incomparable trust.
- **Guarding against memorized case knowledge.** Claude Code named the incident in 21 of 31 transcripts despite the names being withheld, raising the question of how to keep benchmark tasks from being answerable from pretraining rather than from retrieved records.
- **Using verdicts as training signals.** The paper explicitly frames the verifier's verdicts as potential training signals, but does not report any training experiments.

## Target Audience

Researchers and practitioners working on LLM agents and agent evaluation, blockchain forensics and anti-money-laundering analysts, and benchmark designers interested in verifiable-reward task construction. It is most useful to readers who already understand basic fund-tracing mechanics and agent harness design; those seeking a first introduction to blockchain analytics will find the formal definitions dense but the experimental narrative accessible.

## Authors’ abstract

Blockchain forensics is adversarial and open-ended: money launderers split, swap, and bridge funds, cut their trail at mixers or custodial services, and change tactics from one incident to the next. Investigators must keep adapting their tracing in response. Yet the immutability of public ledgers preserves every recorded movement in public view, making much of the investigation independently verifiable. We introduce ABF-Bench, an interactive benchmark that makes…

[The full abstract on arXiv](https://arxiv.org/abs/2610.23656)

[Read the original paper](https://arxiv.org/abs/2610.23656)

---

Canonical: https://crypto.world/paper/abf-bench-a-verifiable-benchmark-for-agentic-blockchain-forensics-2610.23656
