Research
CryptoFlowBench: Benchmarking Multi-Hop Reasoning on Blockchains for Cryptocurrency Fund Tracing
Public blockchains record every transfer but do not link addresses to real identities, so investigators trying to follow stolen funds must reconstruct a suspicious path hop by hop from limited case…

- arXiv
- 2610.16449
- Published
- 2026-10-12
- Authors
- Runang He, Tongya Zheng, Jing Cao, Yucheng Ying, Shunyu Liu, Bingde Hu, Yang Gao, Zeyuan Hu, Jianrong Zhang, Mingli Song
AI summary
Overview
Research area: Blockchain security and cryptocurrency forensics, specifically benchmark design and LLM-agent methods for tracing illicit fund flows across Ethereum transaction graphs.
Technical level: Advanced. The paper assumes familiarity with graph neural networks, taint-propagation heuristics, LLM agent loops (ReAct-style tool use), and standard precision/recall/F1 evaluation.
Scope: The paper introduces CryptoFlowBench, a case-grounded benchmark of 22 documented Ethereum incidents organized into four tracing tasks, together with ChainSleuth, a tool-augmented LLM agent evaluated against heuristic, machine-learning, and GNN baselines.
What This Paper Is About
Public blockchains record every transfer but do not link addresses to real identities, so investigators trying to follow stolen funds must reconstruct a suspicious path hop by hop from limited case evidence while ignoring irrelevant transfers. Existing tools either make independent local predictions that cannot correct an early mistake, or apply fixed propagation rules that sprawl into unrelated but heavily connected services. The paper builds a benchmark that measures this full multi-hop investigation process and proposes an LLM agent that queries on-chain evidence, revisits its earlier decisions, and stops at evidential boundaries.
Key Contributions
-
CryptoFlowBench, a case-grounded benchmark for Ethereum fund-flow investigation. The authors align heterogeneous public incident reports with locally verified on-chain transactions to build case-centered evidence graphs. The benchmark covers 22 incidents across five incident types and derives four complementary tasks under a unified protocol, with strict separation between public task records and oracle-only annotations.
-
ChainSleuth, an evidence-grounded, tool-augmented LLM agent. Within a bounded ReAct loop, the agent maintains an evolving investigation state, selects explicit tracing actions over the space {follow, inspect, stop, ignore}, and retrieves auditable transaction-level evidence through external tools, allowing it to revise earlier hypotheses and suppress unsupported branches.
-
A four-task decomposition that separates different investigative skills. The tasks are transaction existence (Trans), next-step action (Action), path completion (Path), and subgraph filtering (Subgraph), covering factual verification, seeded local decisions, sequential reconstruction under accumulated error, and seedless suspicious-region discovery.
-
A comprehensive, subtask-specific evaluation. ChainSleuth is compared against rule-based taint methods (Poison, Haircut, FIFO, LIFO, TIHO), graph-based tracers (TRacer, DenseFlow/DenseFlow+ adaptations), classical classifiers (LR, RF, MLP, XGBoost), and GNNs (GCN, I²BGNN, PEAE-GNN, TokenScout) under matched candidate nodes, case-disjoint splits, and inference budgets.
Main Findings
-
No baseline dominates all tasks. The experiments show substantial gaps between local fraud detection and multi-hop fund tracing; conventional classifiers stay competitive on single-step decisions but do not translate that strength into end-to-end tracing.
-
ChainSleuth leads on multi-hop Path tracing. With Qwen3.6-27B, ChainSleuth reaches a complete-path rate of 0.665 and Path F1 of 0.734 on the 51 Path tasks from 16 cases, versus the strongest conventional results (LR at 0.432 Path F1, FIFO at 0.422 complete-path rate). This corresponds to a 70.0% Path F1 improvement over the strongest baseline for that metric.
-
Over-expansion is reduced. ChainSleuth lowers over-expansion to 0.116, compared with 0.158 for the strongest baseline — a 26.6% reduction — indicating it can terminate irrelevant branches before errors accumulate along the trajectory.
-
Seedless tracing favors the agent, seeded tracing favors classifiers. On the Subgraph task, Qwen3.6-27B obtains the highest predicted-seed F1 of 0.375 and the highest path recall of 0.568, a 21.4% improvement in predicted-seed F1 over the strongest baseline. Random Forest is stronger when the seed is given (oracle-seed F1 of 0.588, path precision 0.553) but drops to 0.309 predicted-seed F1.
-
Backend gaps track execution reliability, not just reasoning ability. On Path, Qwen3.6-27B achieves a structure-valid rate of 1.000 and a tool-protocol pass rate of 0.979, versus 0.318 and 0.099 for DeepSeek-V4-Flash and 0.780 and 0.292 for DeepSeek-V4-Pro. The same ordering appears on Subgraph, where predicted-seed F1 rises from 0.094 (DS-V4-Flash) to 0.213 (DS-V4-Pro) but stays below 0.375 (Qwen3.6-27B).
-
Detector tools cut both ways. Adding a detector tool (LR, RF, XGBoost) raises Path completeness from 0.665 to 0.760 but lowers Path F1 from 0.734 to 0.628 and raises over-expansion from 0.116 to 0.221; on Subgraph it degrades all reported metrics, suggesting detector scores can bias the agent toward locally plausible but globally inconsistent expansions.
-
The shared seed detector is a fair but not decisive choice. In case-disjoint tests across 1%, 5%, and 10% illicit-node training rates, RF and XGBoost rank comparably (for example AP of 0.881 and 0.883 at 1%) and both beat LR (0.651 AP at 1%), so XGBoost was used for predicted-seed evaluation with oracle-seed results reported separately.
-
Benchmark composition is heavily skewed toward verification. Of 39,184 total task instances, Trans accounts for 38,548 (98.4%), Subgraph 397 (1.0%), Action 188 (0.5%), and Path 51 (0.1%).
-
Motivating scale of the problem. The paper cites the FBI's 2025 Internet Crime Report stating that cryptocurrency was involved in approximately $11 billion of nearly $20 billion in reported losses.
-
Trans and Action results are not in the visible content. The paper states that these diagnostic tasks are reported in Appendix C; the main text provided only includes Path and Subgraph tables.
Methodology in Plain English
The authors start from 22 publicly documented cryptocurrency incidents and organize them into five types: credential/UI compromise, protocol exploit, oracle manipulation, backdoor, and wallet compromise. Web-enabled LLM agents collect reports from disclosures, postmortems, security analyses, and block explorer pages across multiple passes, and a summarization agent distills and verifies them by source priority.
Those summaries are converted deterministically into an evidence specification listing reported addresses, contracts, transfers, roles, and provenance. The authors then query a local blockchain index to pull a bounded k-hop neighborhood around those reported entities, producing a case graph where report-derived semantics are tied to verified on-chain transactions. Absence from the bounded expansion is explicitly not treated as proof of non-existence; exact negatives come only from complete index queries.
From each labeled case graph, a deterministic generator builds instances for the four tasks. Every relation gets an evidence label and one of four actions: follow (continue along the verified flow), inspect (relevant but insufficient evidence), stop (evidential or operational boundary), and ignore (irrelevant). Each instance is split into a public record with only the question, candidates, permitted context, tool declarations, budget, and response schema, and a disjoint oracle record holding answers and gold labels. Oracles never appear in prompts, retrieval stores, or tool outputs.
ChainSleuth then runs a bounded loop: at each step it either calls an allowed tool, receives a sanitized observation, and appends both to its history, or it terminates with a structured JSON answer. Stopping conditions are answer production, evidence coverage, lack of progress, or exhausting the tool budget. For Action, Path, and Subgraph it predicts an action distribution per candidate relation and only expands relations labeled follow above a declared threshold; Subgraph additionally estimates a seed distribution before applying the same edge policy. Evaluation is macro-averaged across incidents with case-disjoint train/validation/test splits so no case-specific information leaks between splits.
Why This Matters
Impact on research: The benchmark turns fund tracing from a static node-classification problem into a sequential, evidence-grounded investigation problem with measurable boundary control. By releasing a case-grounded dataset with strict public–oracle separation and deterministic generation, it gives researchers a reproducible way to compare heuristics, GNNs, and agents on the same footing — something the paper argues neither binary detection benchmarks nor ad hoc case studies provided.
Real-world applications:
- Exchange and bridge incident response: tracing funds after exploits such as those in the benchmark (for example Ronin, Nomad Bridge, Euler Finance, Bybit) and reconstructing which branches are supported by evidence.
- Law-enforcement and forensic casework: producing auditable, transaction-level evidence trails that can be reviewed, rather than opaque model scores.
- Investigative decision support: the stop/ignore action interface explicitly models when an investigator should halt or exclude a branch, which maps to case documentation and reporting.
- Compliance and analytics tooling: helping blockchain-analytics vendors move from fixed taint rules to adaptive, revisable tracing with controlled expansion.
Industry relevance: The paper is directly relevant to blockchain analytics firms, exchange security teams, bridge and DeFi protocol operators, and regtech vendors. Its finding that a detector tool can improve path completeness but degrade precision and over-expansion is a practical caution for teams tempted to bolt classifiers onto agent pipelines.
Future Directions
-
Strengthen the sequential task set. Path contains only 51 instances (0.1% of all instances) drawn from 16 cases and Subgraph evaluation uses a valid-seed, positive-path subset of 22 tasks from 22 cases; expanding these core tracing tasks is a clear next step for statistical robustness.
-
Make agent execution more reliable. The large gap between Qwen3.6-27B (tool-protocol pass rate 0.979) and DeepSeek backends (0.099 and 0.292) suggests that protocol compliance, structured-output validity, and tool-call discipline are open engineering problems for agentic forensics.
-
Design better tool integration. Since detector tools raised completeness but lowered F1 and raised over-expansion on Path and degraded all Subgraph metrics, how to expose classifier signals to an agent without inducing locally plausible but globally inconsistent expansions remains unresolved.
-
Extend beyond the current setting. The benchmark is built on Ethereum incidents within bounded block windows and supported transfer types, leaving open how the tasks and the agent generalize to other chains, cross-chain bridges, and longer tracing horizons.
Target Audience
Researchers and practitioners in blockchain security, cryptocurrency forensics, and graph learning who want a rigorous evaluation setting for multi-hop fund tracing; LLM-agent researchers interested in tool-augmented reasoning over structured graphs with explicit stopping and boundary control; and engineers at analytics vendors, exchanges, and compliance teams who need to judge whether agentic approaches actually outperform the heuristics and classifiers they currently deploy. Readers should be comfortable with GNN baselines and agent-loop terminology, though the problem framing and results are accessible to investigators without a machine-learning background.
Authors’ abstract
Pseudonymous accounts and complex smart-contract ecosystems substantially hinder the tracing of illicit cryptocurrency fund flows. Existing Graph Neural Networks (GNNs) have achieved promising performance in blockchain fraud detection, yet their static node- or edge-level predictions do not naturally generalize to multi-hop fund tracing. Meanwhile, existing tracing methods mainly rely on predefined heuristics to capture multi-hop relationships, limiting their adaptive…
The full abstract on arXiv