Research
LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana
Paired vulnerable/secure Solana bridge programs with labelled attack and benign sessions let auditors measure whether LLM diagnosis actually beats alarm-raising.
- arXiv
- 2611.21777
- Published
- 2026-11-16
- Authors
- Yasir Rahman, Ishraq Samiha, Shahnewaz Karim Sakib, Anindya Bijoy Das
AI summary
Overview
Research area: Blockchain security and AI-assisted smart-contract auditing — specifically, whether open-weight large language models can act as validators for cross-chain bridge programs on Solana.
Technical level: Intermediate. The paper assumes familiarity with Solana accounts, signer privileges, and token programs, but its central argument (an alarm is not a diagnosis) is accessible to anyone who follows blockchain security.
Scope (one sentence): The authors build one native Rust lock-and-mint bridge with four planted, individually verified Solana vulnerabilities plus a secured baseline, then measure how six configurations of five open-weight LLMs perform both as source-code auditors and as transaction-record monitors across 16,800 responses.
What This Paper Is About
Cross-chain bridges concentrate large amounts of user assets, and on Solana many exploits succeed not by breaking consensus but by exploiting a missing application-level check — for example, accepting a token account whose mint was never approved, or trusting a caller-supplied instructions sysvar. The paper asks whether off-the-shelf open-weight LLMs, given only a program's source code or a single session's transaction record, can actually diagnose those specific missing checks rather than merely raising vague alarms. To answer this, the authors build a controlled testbed with known ground truth and separate "the model said yes" from "the model named the planted bug."
Key Contributions
-
A reproducible vulnerable-bridge testbed. Four exploitable Solana bridge variants — A1 mint confusion (modeled on Cashio), A2 sysvar spoofing (modeled on Wormhole), B1 missing signer check, and B2 arbitrary cross-program invocation — plus a secured baseline. Each exploit is verified to succeed against its variant and to be rejected by the secure bridge with a specific error (
WrongMint,UnsupportedSysvar,MissingSignature,WrongTokenProgram). The native Rust implementation, exploit scripts, and transaction traces are released. -
A dual-role evaluation across source code and transaction records. Six configurations of five open-weight models (gpt-oss-120b, Nemotron-3-Super-120B-A12B, Llama-3.3-70B, DeepSeek-V3.2 with thinking disabled and with thinking enabled, and Gemma-4-31B) review inputs 100 times each, producing 12,000 responses across the two primary tracks and 16,800 in total once the comment experiment is added.
-
A blinded diagnosis-grading protocol. Two independent LLM graders (Claude Opus 5.5 and Gemini 3.8 Flash) label 7,200 vulnerable-input responses against reference descriptions of the planted flaws, agreeing on 7,119 (98.88%), with 81 disagreements resolved by a predefined rubric. Labels and decisions are released.
-
A comment-sensitivity analysis. The same five programs are re-evaluated with their original comments restored, holding everything else fixed, to show how vulnerability-revealing commentary inflates measured source-code performance.
Main Findings
-
High alarm rates, low diagnostic accuracy on source code. Without comments, five of six configurations flagged 97–100% of vulnerable source inputs (gpt-oss-120b reached 79%), yet models correctly diagnosed the planted flaw in only 574 of 2,400 runs (24%). Flagged accuracy for those five configurations sat at 50–52%, matching an always-positive detector, which the paper notes would score recall 1.00, F1 0.67, and accuracy 0.50 on this balanced dataset.
-
False positives dominate the secure control. Those configurations produced 373–400 false positives across 400 secure-bridge evaluations, a false-positive rate of 93.25–100%.
-
Comments inflate measured performance. Restoring original comments — some of which describe the flaw — raised final correct source-code diagnoses from 574 to 1,433 of 2,400 graded responses, the basis for the abstract's shift from 24% to 60%.
-
Attack type matters more than model choice in places. On source code, A2 (sysvar spoofing) was the most frequently diagnosed, reaching 73 of 100 runs for DeepSeek-V3.2 with thinking disabled. B1 (missing signer) was rarely identified from source, with no configuration exceeding 15 of 100 runs — the authors attribute this to the retained admin-key comparison making the access-control logic look complete.
-
Transaction monitoring is inverted. From records, five configurations diagnosed B1 (missing signer) in 96–100 of 100 runs, but F1 stayed at 0.67–0.69 because legitimate signed admin rotations drew almost as many alarms. On the other three scenarios, most configurations managed only 0–8 of 100 diagnoses, and the abstract reports an average of 6% across those three.
-
Reasoning mode helps records, not source. Enabling thinking for DeepSeek-V3.2 moved source-code diagnoses only from 98 to 99 of 400, but raised diagnoses on the three harder transaction scenarios from 4 to 91 of 300 runs. Gemma-4-31B showed no comparable gain, so the authors caution that thinking alone does not explain the difference.
-
Best and worst configurations diverge sharply. gpt-oss-120b achieved the highest source diagnosis count (218 of 400) and withheld alarms on roughly one-third of secure-bridge runs, yet its flagged accuracy was still 58%. Llama-3.3-70B identified the planted vulnerability in only one of 800 runs across both tracks and reported reentrancy in
redeem, a vulnerability the Solana runtime prevents and which is absent from the program. -
Monitors respond to failure rather than exploitation. Llama-3.3-70B flagged all 400 blocked attempts but only 100 successful attacks; Nemotron-3-Super flagged 385 blocked attempts versus 226 successful attacks — suggesting some models key on failed transactions rather than evidence of a successful exploit.
-
Batch grading is reliable. Grader agreement was 99.96% on source without comments, 99.38% with comments, and 97.29% on transaction records, the lowest of the three.
-
Diagnosed recall stays low throughout. Across configurations it ranged from 0.00–0.55 on source code and 0.00–0.48 on transaction records.
Methodology in Plain English
The authors did not try to detect real attacks in the wild; they built a controlled laboratory. They implemented a lock-and-mint bridge in native Rust using solana-program v5 and spl-token v9, running it on solana-test-validator (Solana CLI 4.3.0). They then made four near-copies of it, each with exactly one check removed, and wrote a TypeScript exploit for each. Every exploit was confirmed to succeed against its variant and to fail against the secure bridge with a named error, so the ground truth is verified rather than assumed.
For each attack they captured three kinds of session: the successful exploit, a benign operation, and a blocked attempt rejected by the secure bridge. The benign case for the missing-signer attack is an authorized admin rotation — a deliberate matched negative, since both the attack and the legitimate action change the admin key.
Models were then given one of two inputs and asked three plain questions. Source-code track: does this program contain a vulnerability, name the class, point to the function and missing check. Transaction track: does this trace show an attack, what is suspicious, name the likely class. To prevent leakage, source comments were stripped in the main condition, record labels were anonymized, and no attack classes, function names, bug counts, or safety labels were provided. Models ran at temperature 0 with a 16,000-token output limit through DeepInfra, 100 times per input.
Responses were scored at two levels: Flagged (did the model say yes) and Diagnosed (did it name the planted mechanism and, for source, the correct function). Only "yes with the planted bug named" counts as a true positive at the diagnostic level; a yes with an unrelated explanation is a detection success but a diagnostic miss. Scores use 400 vulnerable and 400 secure or benign runs per configuration, with blocked sessions excluded. The RQ3 comment experiment repeats the source evaluation with comments restored; the provided text gives its headline totals but not the full per-model breakdown.
Why This Matters
Impact on research. The paper undermines a common proxy in LLM security evaluation: treating "the model flagged it" as evidence of detection capability. Because five configurations flag nearly everything, their apparent 97–100% recall is indistinguishable from a trivial always-yes baseline. Separating alarm from diagnosis, adding matched negatives (a legitimate admin rotation alongside the attack), and documenting comment handling gives the field a template for evaluations that cannot be passed by a detector that never says no.
Real-world applications:
- Pre-deployment audit triage for bridge programs — indicating where LLM review can assist and where it currently cannot substitute for human auditors, particularly for authorization checks that look intact because adjacent validation logic remains.
- Post-incident and real-time monitoring for bridge operators — the finding that models confuse legitimate admin key rotations with unauthorized ones directly informs how alerting thresholds and context should be designed, and why alert volume alone is not a security signal.
- Vendor and tool procurement — teams evaluating "AI auditing" products get concrete metrics (diagnosed recall of 0.00–0.55 on source) to demand instead of flag rates.
- Benchmark design for smart-contract LLM security claims — the four-variant, paired-vulnerable-and-secure design with verified exploits is reusable for other chains and other vulnerability classes.
Industry relevance. The motivating losses are large and recent: Chainalysis estimated approximately $2 billion stolen across 13 bridge hacks by August 2022, with bridge attacks accounting for 69% of cryptocurrency theft that year to date. Wormhole's forged sysvar account enabled 120,000 unbacked wrapped ETH, worth approximately $320 million, and Cashio lost approximately $52.8 million. Both are reproduced here as mechanisms, not full systems, and the paper's point is that neither required breaking the blockchain — only a missing application check.
Future Directions
- Combining source and transaction evidence. No run in this study gives a model both inputs; the authors explicitly keep the auditor and monitor roles separate, leaving open whether joint evidence closes the gap — especially for the missing-signer case, where source is hard and records are ambiguous.
- Closing the authorization-check blind spot. B1 was diagnosed in under 15 of 100 source runs by every configuration, and transaction-based detection was confounded by benign admin rotations. Whether targeted prompting, retrieval of audit knowledge, or fine-tuning resolves this is untested here.
- Reasoning-mode attribution. DeepSeek-V3.2 with thinking rose from 4 to 91 of 300 on the three harder transaction scenarios while Gemma-4-31B showed no such gain; the paper leaves open what actually drives the difference rather than attributing it to thinking alone.
- Standardizing comment and prompt handling. Since restoring original comments moved correct diagnosis from 574 to 1,433 of 2,400, the authors argue comment handling must be documented when interpreting source-code performance — a norm that does not yet exist and would change reported numbers across the literature.
Target Audience
Bridge and Solana protocol developers who need to know what current LLM review can and cannot catch; smart-contract auditors assessing AI assistance in their workflow; LLM evaluation researchers interested in separating detection from diagnosis and in using matched negatives and blinded grading; and security, risk, and monitoring teams at firms operating or insuring cross-chain infrastructure who need realistic expectations about automated alerting.
Authors’ abstract
Cross-chain bridges have suffered substantial asset losses due to vulnerabilities in their on-chain programs. On Solana, missing validation checks can enable exploits even when the underlying blockchain operates correctly. The ability of large language models (LLMs) to diagnose these flaws from source code or transaction records, rather than merely raise alarms, remains insufficiently understood. We implement a native Rust lock-and-mint bridge on a local Solana validator…
The full abstract on arXiv