Skip to content
Crypto.world

Research

Prompt Engineering vs. Fine-Tuning for LLM-Based Vulnerability Detection in Solana and Algorand Smart Contracts

Most smart contract vulnerability research and tooling targets Ethereum and Solidity, leaving newer platforms such as Solana (Rust) and Algorand (PyTeal/TEAL) largely unexamined.

Prompt Engineering vs. Fine-Tuning for LLM-Based Vulnerability Detection in Solana and Algorand Smart Contracts
arXiv
2511.11250
Published
2025-11-14
Authors
Biagio Boi, Christian Esposito

AI summary

Overview

Research area: Blockchain smart contract security, specifically LLM-based static vulnerability detection for non-EVM platforms (Solana and Algorand), with a comparison of prompt engineering, fine-tuning, and a hybrid of the two.

Technical level: Intermediate. Readers benefit from basic familiarity with smart contracts, LLM prompting concepts, and fine-tuning terminology, but the paper explains both.

Scope (one sentence): The paper builds a synthetic, OWASP-derived dataset of Rust (Solana) and PyTeal (Algorand) contract snippets and benchmarks two open-source LLMs — LLAMA-3-8B and DeepSeek-R1-Distill-Qwen-14B — under prompt engineering, fine-tuning, and combined configurations.

What This Paper Is About

Most smart contract vulnerability research and tooling targets Ethereum and Solidity, leaving newer platforms such as Solana (Rust) and Algorand (PyTeal/TEAL) largely unexamined. Because no labeled vulnerability datasets exist for these non-EVM platforms, the authors construct their own annotated snippets and test whether general-purpose LLMs can detect platform-specific vulnerabilities without and with adaptation. The goal is to determine which adaptation strategy — prompting, fine-tuning, or both — actually works for each language and vulnerability class.

Key Contributions

  1. A platform-specific vulnerability mapping. The ten OWASP Top 10 (2025) categories are mapped onto Algorand and Solana, showing which ones translate directly, which appear under different names (e.g., Bump Seed, Unchecked RekeyTo, Type Confusion), and which do not apply at all.

  2. A synthetic, balanced dataset for non-EVM platforms. Annotated snippets in Rust for Solana and PyTeal for Algorand were built manually around the OWASP-derived taxonomy, with at least five vulnerable and five non-vulnerable instances per vulnerability type, supported by symbolic and structural analysis.

  3. A three-way empirical comparison. Two open-source models (LLAMA-3-8B and DeepSeek-R1-Distill-Qwen-14B) are evaluated under fine-tuning, role-based prompt engineering, and the combination, using accuracy, precision, recall, and F1-score.

  4. An architectural analysis of detectability. The paper explains why the same vulnerability class is easier or harder to detect depending on the platform's execution model and language expressiveness.

Main Findings

  • DeepSeek outperforms LLaMA in the unmodified baseline. Without any prompt engineering or fine-tuning, DeepSeek reached average accuracy of 0.63 on Solana versus LLaMA's 0.53, and 0.60 on Algorand versus LLaMA's 0.50.

  • DeepSeek is precise but inconsistent. On Solana it reached precision of 1.00 on most categories (0.60 for CPI), but recall ranged from 0.20 (Type Confusion) to 0.60 (Bump Seed) — it rarely flagged vulnerable code as vulnerable.

  • LLaMA fails completely on Algorand without adaptation. LLaMA produced a null F1-score across every Algorand category in the baseline, with zero recall, indicating an inability to capture PyTeal vulnerability patterns out of the box.

  • Prompt engineering alone is a strong, cheap baseline for Solana. DeepSeek's best Solana result came from role-based prompt engineering alone, at 0.63 average accuracy — matching or exceeding its fine-tuned result (0.57) and its combined result (0.60).

  • Fine-tuning helps most on the less expressive language. DeepSeek's best Algorand result came from fine-tuning (0.62 average accuracy), compared with 0.54 for prompt engineering alone.

  • LLaMA peaks with the hybrid approach. LLaMA's highest score across all configurations was 0.65 average accuracy on Algorand when prompt engineering was combined with fine-tuning.

  • Algorand is architecturally more resistant. Four of the ten OWASP categories are not applicable to Algorand, due to its atomic, stateless transaction model and lack of dynamic callbacks or inter-contract call stacks. All ten remain theoretically applicable on Solana, though practical exploitability is constrained by the parallel execution model.

  • Only two categories map one-to-one across all three platforms. Access Control (V1) and Unchecked External Calls (V6) hold a clear correspondence across Ethereum, Algorand, and Solana, which the authors read as evidence that a uniform security model cannot be transferred wholesale to non-EVM chains.

  • Detection difficulty tracks how explicitly a flaw is encoded. Easier cases such as Missing Key Check (Solana) and Unchecked Asset Receiver (Algorand) scored high across configurations, while subtler logic-dependent cases such as Unchecked Close Remainder and Type Confusion were highly variable. The authors report V1 and V10 as the easiest categories to detect overall.

  • A note on reported data: the paper states DeepSeek achieved perfect precision on all five Solana vulnerabilities, while Table III lists 0.60 precision for CPI. The paper does not report the total dataset size in absolute terms, nor any cost or latency figures.

Methodology in Plain English

The authors first read across the two blockchains' architectures and asked, for each of the ten OWASP vulnerability categories, whether it exists on Solana and Algorand and what it is called there. They then hand-wrote small contract snippets for each relevant vulnerability — five that contain the flaw and five that do not — in Rust for Solana and PyTeal for Algorand, giving a balanced labeled set per category.

Each snippet was passed to a freshly loaded model with a short, neutral system prompt asking whether the code contains a vulnerability, so no answer was influenced by a previous one. Answers were treated as binary classifications: correctly identifying the target vulnerability counted as a true positive; naming a different vulnerability counted as a false negative. Every prompt was run three times per sample and averaged to reduce randomness, giving 30 requests per vulnerability type.

The same snippets were then re-tested under a role-based prompt ("You are a smart contract security analyzer… classified according to OWASP Top 10"), under LoRA fine-tuning, and under both together. Fine-tuning used the Hugging Face transformers and peft libraries with LoRA rank 64, alpha 16, dropout 0.1, an Adam optimizer at learning rate 2e-4, and a maximum sequence length of 2048 tokens; DeepSeek was trained for 10 epochs and LLaMA for 4. A post-processing rule mapped generated tokens to binary labels.

Why This Matters

Impact on research: The paper extends LLM-based vulnerability detection beyond the Solidity/Ethereum setting that dominates the literature, and it provides a reusable mapping of OWASP categories onto two structurally different chains. It also gives a controlled comparison of adaptation strategies, which is often asserted but rarely measured for domain-specific languages.

Real-world applications:

  • Pre-deployment screening in CI pipelines for Solana programs and Algorand applications, where a cheap prompt-based check can run before a formal audit.
  • Audit triage: using precision-oriented configurations to surface candidate findings for human reviewers, particularly for access control and unchecked-call issues.
  • Security tooling for non-EVM platforms, where Ethereum tools such as SmartCheck and Slither cannot be applied directly due to different languages and execution models.
  • Developer education, by mapping named vulnerability patterns (Bump Seed, Unchecked RekeyTo, Arbitrary Delete) onto recognizable code anti-patterns.

Industry relevance: The results argue against assuming a single LLM setup works everywhere. Teams on Rust-based chains can get competitive results from prompting alone, while teams on stack-based languages like TEAL should budget for fine-tuning or a hybrid pipeline. The finding that LLaMA scores zero recall on Algorand without adaptation is a warning against deploying a general model unmodified and trusting its silence.

Future Directions

  • Few-shot prompting was not evaluated. The paper used zero-shot and role-based prompting only and explicitly names few-shot techniques as future work, so the upper bound of prompt engineering here remains unmeasured.
  • Larger datasets. The current set is small (five vulnerable and five non-vulnerable snippets per category), which the authors acknowledge limits precision of evaluation and raises overfitting concerns for fine-tuning.
  • Generalization beyond two models and two chains. Whether these patterns hold for proprietary models such as GPT-4, which the authors excluded due to incompatibility with fine-tuning, or for other non-EVM ecosystems, is untested.
  • Turning the vulnerability mapping into tooling. The paper documents platform-specific mappings but does not build an analyzer; operationalizing the taxonomy into a static analysis or detection pipeline is an open engineering question.

Target Audience

Blockchain security researchers and auditors working outside the Ethereum ecosystem; LLM practitioners interested in domain adaptation for code; smart contract developers on Solana and Algorand who want to understand which vulnerability classes actually apply to their platform; and tooling vendors deciding whether to invest in prompting or fine-tuning for non-EVM chains. Readers seeking production-ready detection numbers or cost analysis will not find them here — the contribution is comparative and taxonomic rather than a deployed system.

Authors’ abstract

Smart contracts have emerged as key components within decentralized environments, enabling the automation of transactions through self-executing programs. While these innovations offer significant advantages, they also present potential drawbacks if the smart contract code is not carefully designed and implemented. This paper investigates the capability of large language models (LLMs) to detect OWASP-inspired vulnerabilities in smart contracts beyond the Ethereum Virtual…

The full abstract on arXiv
Read the original paper