# Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation

Crypto.world

> Most LLM-based smart contract auditors rely on one large general-purpose model that must handle the entire audit in a single prompt, which is expensive, memory-hungry, and usually stops at binary…

**arXiv**: 2606.03128

**Published**: 2026-06-02

**Authors**: Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra

## AI summary

## Overview

**Research area:** Blockchain security / smart contract auditing using large language models (LLM-assisted vulnerability detection), with a focus on lightweight on-device inference.

**Technical level:** Intermediate. Readers should be comfortable with fine-tuning concepts (LoRA, supervised fine-tuning), the idea of knowledge distillation, and standard classification metrics (precision, recall, F1, accuracy), but the paper explains its architecture in accessible terms.

**Scope (one sentence):** The paper proposes and evaluates a four-stage, decoupled smart contract audit pipeline built on 0.6B–4B parameter open-source Qwen3 models, trained with rsLoRA, knowledge distillation and a modified Chain-of-Verification aggregation step, and benchmarks it against 7B–34B dense coder LLMs.

## What This Paper Is About

Most LLM-based smart contract auditors rely on one large general-purpose model that must handle the entire audit in a single prompt, which is expensive, memory-hungry, and usually stops at binary "vulnerable / not vulnerable" output without severity ranking or fix advice. The authors argue that a small, task-specialized model per audit stage can beat much larger general models while running on ordinary consumer hardware. Their goal is an end-to-end pipeline that detects a vulnerability, explains how it can be exploited, assigns a severity level, and recommends a concrete remediation.

## Key Contributions

1. **A decoupled four-stage audit framework.** The audit is split into a Vulnerability Detector, a Vulnerability Explanator, a Severity Determinant and a Recommender, each passing context sequentially to the next, replacing a single monolithic prompt.
2. **Reasoning distillation plus a modified Chain-of-Verification aggregator.** For the two generative stages (explanation and recommendation), the authors distill chain-of-thought reasoning from a Qwen3-30B-A3B teacher into a Qwen3-4B student, then use a custom CoVe aggregator that verifies and consolidates five independently generated candidate drafts instead of self-verifying a single draft.
3. **A first in-depth study of LLM severity classification for smart contracts.** The authors report that severity grading is substantially harder than binary detection and identify a "severity centrality bias," where models disproportionately default to "Medium" and fail on Low and High edge cases.
4. **Ablation-driven evidence that small beats large here.** The paper reports that its 0.6B–4B pipeline outperforms fine-tuned and zero-shot 7B–34B dense coder baselines on every evaluated task, and quantifies the contribution of rsLoRA, voting, distillation and CoVe separately.

## Main Findings

- **Vulnerability detection:** The proposed pipeline (rsLoRA + 5-prompt voting) reaches **98.25% accuracy**, precision **0.9829**, recall **0.9822** and F1 **0.9824**. The best fine-tuned baseline (Qwen2.5-Coder-14B) reaches F1 **0.9684** and accuracy **0.9684**; the best zero-shot baseline (Qwen2.5-Coder-32B) reaches F1 **0.7389** and accuracy **0.7404**.
- **Detection ablations:** Removing voting from the rsLoRA model drops F1 to **0.9754**. Swapping rsLoRA for standard LoRA while keeping voting drops F1 to **0.9509**; standard LoRA without voting gives F1 **0.9684**.
- **Vulnerability explanation:** The full pipeline scores **0.4375** on the LMUnit 70B alignment judge. Best fine-tuned baseline is Qwen2.5-Coder-14B at **0.284**; best zero-shot is DeepSeek-Coder-33B at **0.229**.
- **Explanation ablations:** Modified CoVe without distillation scores **0.3055**; distillation without CoVe scores **0.2569**; plain rsLoRA fine-tuning with neither scores **0.1111**. The authors describe the combination as synergistic.
- **Recommendation generation:** The full pipeline scores **0.763**, versus **0.479** for the best zero-shot baseline (Codestral-22B) and **0.375** for fine-tuned Qwen2.5-Coder-14B and Qwen2.5-Coder-32B.
- **Recommendation ablations:** Standard LoRA instead of rsLoRA gives **0.625**; CoVe with rsLoRA only gives **0.597**; CoVe with standard LoRA gives **0.423**; distillation with rsLoRA only gives **0.270**; distillation with standard LoRA gives **0.333**; basic LoRA **0.263**; basic rsLoRA **0.187**.
- **Severity classification:** The Severity Determinant achieves macro F1 **0.5877** and accuracy **64.58%** (precision **0.6197**, recall **0.5745**), ahead of all baselines. The strongest baseline on accuracy is fine-tuned CodeGemma-7B at **0.6111** (macro F1 **0.4332**).
- **Severity centrality bias (SCB):** Both zero-shot and fine-tuned open-source models tend to over-predict "Medium" severity and struggle to distinguish Low from High cases, making severity a distinctly harder task than binary detection. The paper's extended discussion of this phenomenon is cut off in the provided text.
- **rsLoRA matters:** The authors attribute part of the gain to the rank-stabilized scaling factor (α/√r instead of α/r), which they state avoids the gradient collapse that standard LoRA shows as rank increases.
- **Computational cost:** Cost is measured on a **V100 GPU with 32 GB VRAM** using GPU utilization and maximum used GPU memory as metrics. The comparative cost figures are not included in the excerpt provided.

## Methodology in Plain English

**Data.** The authors crawled public audit reports from Code4rena and Shieldify Security for vulnerable code, and used the top **500** smart contract projects by Total Value Locked as benign samples. This produced **930** vulnerable and **969** non-vulnerable samples for the detection dataset, and a separate vulnerability dataset with **218** low-, **549** medium- and **232** high-severity findings for the generative stages. Splits are 70/15/15: **1,329 / 285 / 285** for detection, and **710 / 145 / 144** for the vulnerability dataset.

**Avoiding shortcuts and leakage.** Because vulnerable snippets are short and benign files are whole contracts, the authors treat each contract as a Function Call Graph, extract connected sub-graphs as benign snippets, and then sample them so that the snippet-length distribution matches the vulnerable set (minimizing KL divergence). This stops the model from learning "short code = vulnerable." For the vulnerability dataset they split by project identity with iterative stratification so the same project never appears in both training and test sets, and they cross-reference the end-to-end test set against training code to prevent leakage.

**Architecture.** Detection and severity classification use Qwen3-0.6B with rsLoRA, since both are classification problems. Explanation and recommendation use Qwen3-4B with rsLoRA, since they require generation and reasoning. Detection and severity use a voting scheme: five different template prompts are run and the majority answer becomes the output, with a confidence score equal to the fraction of prompts agreeing.

**Distillation and aggregation for generation.** A Qwen3-30B-A3B teacher (a mixture-of-experts model that activates only 3B parameters at inference) generates five chain-of-thought reasoning traces per example, which are combined with the ground-truth answer to fine-tune the Qwen3-4B student. At inference, the student produces five candidate explanations or fixes, and an aggregator with the default thinking mode—trained only as a candidate generator, not the aggregator—runs the modified CoVe loop: ground itself in the contract code, write verification questions for each candidate, answer them independently, then synthesize a single report keeping only verified content and discarding speculative claims. The authors argue this fixes a weakness in standard CoVe, where a single plausible hallucinated draft biases the model's own fact-checking.

**Evaluation.** Classification stages use accuracy plus macro-averaged precision, recall and F1. Generative stages use an LLM-as-a-judge alignment score between 0 and 1, produced by the LMUnit 70B evaluator (from RewardBench-2) with the ATLA-Selene classification-with-reference prompt template, comparing outputs to human-written ground truth. Baselines are CodeLlama-13B/34B, CodeGemma-7B, Codestral-22B, DeepSeek-Coder-7B/33B and Qwen2.5-Coder-14B/32B, each in zero-shot and, where applicable, fine-tuned settings. The teacher model is excluded from baselines because its full weight footprint is impractical on the consumer hardware targeted.

## Why This Matters

**Research impact.** The paper reframes smart contract LLM auditing as a pipeline-engineering problem rather than a scale problem, and provides the first structured evidence that severity grading—not detection—is the bottleneck. The severity centrality bias is a reportable empirical phenomenon that later work can try to correct, and the ablation tables give a clean attribution of gains to rsLoRA, voting, distillation and verification-based aggregation.

**Real-world applications.**

- Local pre-commit or CI hooks that flag vulnerable Solidity in a developer's own environment under an 8–12 GB VRAM budget, supporting "shift-left" security.
- Triage tooling for small audit teams that orders findings by severity so limited reviewer time goes to High-risk issues first.
- Educational and training use, where the explanation stage shows how a flaw can be exploited step by step.
- Patch suggestion for independent Web3 developers and open-source teams who cannot afford paid audits or cloud GPU instances.

**Industry relevance.** The paper targets a concrete cost barrier: cloud GPU instances and 7B–34B dense models are expensive and often exceed memory on entry-level to mid-range consumer hardware (the paper cites 8–12 GB VRAM for standard consumer machines and 16–24 GB for high-end enthusiast GPUs). A framework that runs on a single 32 GB V100 and still outperforms 34B dense baselines changes the economics of who can ship an audited contract. The paper also frames the work against real losses, citing the DAO reentrancy exploit at roughly $60 million.

## Future Directions

1. **Correcting severity centrality bias.** The paper identifies over-prediction of "Medium" severity but the provided text does not describe a mitigation; calibrated or ordinal severity modeling, or severity-aware resampling, is a natural next step.
2. **Pushing below 0.6B or above 4B selectively.** The authors show that small task-specific models win at 0.6B for classification and 4B for generation. Whether the same decoupling holds for multi-file contracts, cross-contract calls, or non-Solidity targets is untested here.
3. **Replacing or shrinking the aggregator.** The aggregator relies on a reasoning-capable model with thinking mode and is not fine-tuned, which is likely the dominant inference cost. Distilling the aggregator itself, or reducing the five-candidate budget, is an open efficiency question.
4. **Independent reproduction and benchmark standardization.** The datasets are assembled by the authors from Code4rena and Shieldify Security plus top-500 TVL projects, and the generative metric depends on a specific LMUnit 70B plus ATLA-Selene prompt configuration. Whether the margins hold under different judges, audit platforms and contract ecosystems is not reported.

## Target Audience

Researchers working on LLM-assisted program analysis and blockchain security; security engineers and auditors who want a practical, local-first triage assistant; ML practitioners interested in small-model distillation, rsLoRA and verification-based aggregation for structured generation tasks; and Web3 developers or small teams who need automated auditing on consumer-grade hardware rather than cloud GPUs.

## Authors’ abstract

Smart contracts face critical security challenges that require thorough auditing in decentralized web services. While Large Language Models (LLMs) have shown promise in automated vulnerability detection, existing approaches lack severity evaluations with actionable remediation and demand unnecessarily massive computational overhead. In this study, we introduce an efficient end-to-end smart contract security audit framework utilizing lightweight, highly optimized open-source…

[The full abstract on arXiv](https://arxiv.org/abs/2606.03128)

[Read the original paper](https://arxiv.org/abs/2606.03128)

---

Canonical: https://crypto.world/paper/decoupled-smart-contract-audits-lightweight-llm-framework-via-2606.03128
