# FLAMES: Fine-tuning LLMs to Synthesize Invariants for Smart Contract Security

Crypto.world

> Existing smart contract vulnerability detectors produce unreliable verdicts and, crucially, do not output anything a developer can actually deploy as a defense.

**arXiv**: 2510.21401

**Published**: 2025-10-24

**Authors**: Mojtaba Eshghie, Gabriele Morello, Matteo Lauretano, Alexandre Bartel, Martin Monperrus

## AI summary

## Overview

**Research area:** Blockchain security and program synthesis — specifically, using domain-adapted large language models to generate executable runtime guards (invariants) for Solidity smart contracts.

**Technical level:** Advanced. The paper assumes familiarity with smart contract semantics, the Solidity `require` statement, abstract syntax trees, parameter-efficient fine-tuning (QLoRA/LoRA), fill-in-the-middle training objectives, and SMT-based semantic equivalence checking.

**Scope in one sentence:** The paper presents FLAMES, a fine-tuned CodeLlama2-7B pipeline that takes a smart contract and a target function, synthesizes Solidity `require` invariants at function entry and exit, and injects them to block exploits — evaluated on compilation success, semantic similarity to human-written invariants, mitigation of 108 reproducible exploits, and one real-world incident.

Note: the supplied paper content is truncated partway through Section V-C, so the threats-to-validity (Section VI), related work (Section VII) and conclusion (Section VIII) sections are not summarized here.

## What This Paper Is About

Existing smart contract vulnerability detectors produce unreliable verdicts and, crucially, do not output anything a developer can actually deploy as a defense. The authors instead synthesize *invariants* — boolean conditions written as Solidity `require(...)` statements at function entry (pre-conditions) and exit (post-conditions) — which revert dangerous transactions at runtime. The goal is a fully automated, label-free pipeline that hardens a real contract using only its source code, with no vulnerability labels, no symbolic analysis, no formal specifications, and no natural-language hints.

## Key Contributions

1. **FLAMES, an end-to-end invariant synthesis pipeline.** A three-stage design — AST-based code analysis and context abstraction, fill-in-the-middle supervised fine-tuning, and inference with pre-/post-condition injection — that hardens contracts using only source code as input.
2. **Disl, a large unique Solidity dataset.** Built from 3,298,271 raw deployed contract records, yielding 12,931,943 decomposed Solidity files and, after Jaccard-similarity deduplication at a 90% threshold, 514,506 unique verified files containing 5,276,315 `require` statements. The paper reports that over 96% of decomposed files were duplicates.
3. **Disl-HardInv, a curated hard test set.** 5,000 unique, non-trivial invariants filtered to exclude ones containing only variable identifiers, simple comparisons, negations, straightforward function calls, or hardcoded addresses, held out from the fine-tuning corpus to prevent data leakage.
4. **A three-dimensional evaluation plus a real-incident case study.** Compilability, semantic equivalence against human-written invariants via the Sindi checker, exploit mitigation on 108 vulnerable contracts from the SB-Heist benchmark with functionality regression tests, and a reproduction of the APEMAGA incident.

## Main Findings

- **Compilation (RQ1): 96.7% of synthesized invariants compile.** FLAMES-100k produced compiling `require` statements in at least 4,835 of 5,000 cases, versus 3,130 of 5,000 for the non-domain-adapted CodeLlama baseline (62.6%). The authors attribute the gap to learned syntactic (scoping) and semantic (typing) constraints of Solidity; the baseline's characteristic failure is emitting a function declaration inside another function body, whereas FLAMES returns a single well-formed invariant at the mask location.
- **Semantic quality (RQ2): 44.5% exact-or-equivalent on Disl-HardInv, versus 23.5% for the baseline.** Exact textual matches rise monotonically with fine-tuning — 959 for CodeLlama, 1,328 for FLAMES-20k, 1,840 for FLAMES-100k — as do semantically equivalent but textually different matches: 217, 362, 386. (The paper's RQ2 answer box attributes 1,328 to the baseline; the detailed Figure 8 discussion attributes 959 to the baseline, which is what the 23.5% figure is consistent with.)
- **Partial semantic alignment grows too.** Cases where the synthesized invariant is either stronger or weaker than ground truth rise from 157 (CodeLlama) to 168 (FLAMES-20k) to 446 (FLAMES-100k). Of those 446, 209 are stronger and 237 weaker.
- **Exploit mitigation (RQ3): best configuration blocks 22 of 108 real exploits (20.4%) with functionality preserved.** Under multi-turn inference, FLAMES-20k blocks 16/108, 20/108 and 22/108 for pre-, post-, and pre+post-condition injection respectively; FLAMES-100k blocks 21/108, 22/108 and 20/108. The CodeLlama-7B baseline reaches 16/108, 10/108 and 13/108.
- **Specialization, not model size, drives mitigation.** FLAMES-20k needs multi-turn pre+post injection to reach its maximum of 22/108, while FLAMES-100k reaches the same 22/108 with a single post-condition — suggesting that post-conditions are often the most effective placement in practice.
- **Effectiveness is strongly vulnerability-class dependent.** For access control (16 contracts), FLAMES-100k reaches 7 across all three injection locations. Arithmetic (20 contracts) benefits most from post-conditions (7, versus 3 for pre and 6 for pre+post). Reentrancy (26 contracts) shows the opposite pattern, with pre-conditions strongest (7 versus 2 and 3). Time manipulation reaches 2/4. Unchecked low-level calls (22 contracts) yield zeros for both FLAMES variants across all placements. Front running (6) and the "Other" (2) class remain at zero for all models and placements.
- **Functionality is preserved far more often than with the baseline.** 41.3% (134/324) of baseline-generated invariants preserve contract functionality, versus 64.5% (209/324) for FLAMES-20k and 67.9% (220/324) for FLAMES-100k.
- **Multi-turn prompting matters mainly for the smaller model.** FLAMES-20k's pre+post results improve from 17/108 (single-turn) to 22/108 (multi-turn); FLAMES-100k is essentially flat (pre: 21/21, post: 22/22, pre+post: 21 versus 20).
- **Real incident (RQ4): the APEMAGA attack is blocked.** APEMAGA was exploited in June 2024 with a loss of approximately 9.4 ETH. FLAMES-100k synthesizes a pre-condition semantically equivalent to the published ground-truth patch; the hardened contract is validated by forking Ethereum mainnet at the block prior to the attack, replacing the token's runtime code at the APEMAGA address while preserving storage, and running the exploit proof-of-concept.
- **Failure modes are characterized, not just counted.** Weakening typically comes from dropping conjuncts, relaxing thresholds, or replacing compound conditions with a single flag (for example, doubling an aggregate fee cap from `<=25` to `<=50`, or omitting a per-block mint cap or a supply bound). Strengthening comes from replacing inequalities with equalities, removing disjunctions, or hard-coding units such as multiplying by `1e18`.

## Methodology in Plain English

**Code analysis.** Every contract is parsed into an abstract syntax tree. Because more than 31% of contracts in Disl exceed 4,096 tokens (roughly 300 lines of Solidity, and the context limit of many base models), the pipeline builds a compact "abstracted" view: the full body of the target function, definitions of functions that call it, applicable modifiers, signatures only for other functions, state variable declarations, and no event definitions. The same abstraction is used at training and inference time.

**Training.** The authors mine real `require(...)` statements from Disl. Each training sample is an abstracted contract with one invariant replaced by a `<FILL_ME>` placeholder; the model must reconstruct the missing predicate from surrounding code — a fill-in-the-middle objective. This requires no vulnerability labels, only existing developer-written invariants.

**Fine-tuning setup.** Base model CodeLlama2-7B in 4-bit quantization (nf4), QLoRA with LoRA rank r=8 and alpha=16 injected into attention Q/V projections, context length 4,096 tokens, AdamW optimizer, learning rate 3×10⁻⁴, batch size 32 (per-device 2, gradient accumulation 16), gradient clipping, one epoch on 4× Nvidia A100 80GB GPUs. Two variants are released: FLAMES-20k (20,000 samples) and FLAMES-100k (100,000 samples).

**Inference.** A user specifies a contract and target function; the pipeline infills `require(<FILL_ME>)` at the chosen location (function entry for a pre-condition, function exit for a post-condition), generates the boolean predicate, substitutes it back, reconstructs the full contract, compiles it, and returns the hardened source. Multiple invariants can be generated single-turn (each prompt sees the original contract) or multi-turn (each prompt carries the previously synthesized invariants).

**Evaluation design.** Compilability is checked by compiling under each contract's original pragma and solc configuration. Semantic quality is judged by Sindi, an SMT-and-rewrite-rule equivalence checker, which classifies each synthesized/ground-truth pair as exact match, semantically equivalent, synthesized stronger, ground truth stronger, or inconclusive. Exploit mitigation is measured on 108 contracts from SB-Heist: a hardening counts as successful only if the contract compiles, passes all of the original functional tests, and reverts the ground-truth exploit. No vulnerability labels are given to the model at any point.

## Why This Matters

**Research impact.** The paper reframes smart contract defense from *detection* to *synthesis of deployable artifacts*. It shows that a fine-tuned 7B model trained purely on existing `require` statements — with no vulnerability supervision at all — can produce compile-clean, sometimes semantically equivalent guards. It also contributes the largest unique verified Solidity corpus reported in the paper's own comparison table, alongside a reusable hard benchmark and an evaluation harness.

**Real-world applications:**

- **Pre-deployment hardening** of DeFi contracts, where developers can run synthesis over sensitive functions and review candidate `require` statements before deployment.
- **Emergency patching of live contracts** through proxy upgrades or migration, generating a candidate guard quickly during an active incident.
- **CI/CD security gates**, adding synthesized invariants as a check that fails when an implementation drifts from intended pre-/post-conditions.
- **Developer education and audit triage**, surfacing invariants a developer may have forgotten on a specific code path, and giving auditors concrete predicates to reason about.

**Industry relevance.** Billions of dollars are lost annually to smart contract exploits, and the paper argues existing detectors neither give reliable verdicts nor produce deployable defenses. A technique that emits ordinary Solidity `require` statements fits existing deployment, auditing and testing workflows without new tooling or formal-methods expertise. The authors release code, model weights (hosted on Hugging Face), datasets and the evaluation harness for reproducibility.

## Future Directions

1. **Cover the vulnerability classes that resist pre-/post-conditions.** Front running (6 contracts) and the "Other" class (2 contracts) stayed at zero for every model and placement, and unchecked low-level calls (22 contracts) stayed at zero for both FLAMES variants across pre, post and pre+post. New invariant forms or additional synthesis locations are needed for these.
2. **Reduce weakening and false positives systematically.** Weakened invariants that drop conjuncts or relax thresholds still compile and may silently fail to stop exploits or over-constrain legitimate behavior. Only 67.9% (220/324) of FLAMES-100k invariants preserved functionality, so a filtering or self-verification stage is a natural next step.
3. **Compose orthogonal properties.** The inconclusive category in Table III reflects invariants over unrelated state dimensions (authorization versus accounting versus allowance toggles). The authors suggest multi-turn prompting or explicit composition when several orthogonal properties must hold jointly.
4. **Test generalization beyond the current base model and benchmark.** Only CodeLlama2-7B is fine-tuned, and only one real incident (APEMAGA) is reproduced. Whether the approach carries to other base models, larger training subsets than 100,000 samples, attack classes absent from SB-Heist, and incidents that are not preventable by a single pre-condition remains open.

## Target Audience

Smart contract security researchers and auditors; blockchain engineers responsible for DeFi contract hardening; applied machine learning researchers interested in domain adaptation and fill-in-the-middle training for code; and tooling engineers building CI/CD security pipelines for Solidity. Readers without a background in smart contract semantics or LLM fine-tuning will find the evaluation design accessible but the mechanics of context abstraction and Sindi-based semantic classification difficult.

## Authors’ abstract

Smart contract vulnerabilities cost billions of dollars annually, yet existing automated analysis tools fail to generate deployable defenses. We present FLAMES, a novel automated approach that synthesizes executable runtime guards as Solidity "require" statements to harden smart contracts against exploits. Unlike prior work that relies on vulnerability labels, symbolic analysis, or natural language specifications, FLAMES employs domain-adapted large language models trained…

[The full abstract on arXiv](https://arxiv.org/abs/2510.21401)

[Read the original paper](https://arxiv.org/abs/2510.21401)

---

Canonical: https://crypto.world/paper/flames-fine-tuning-llms-to-synthesize-invariants-for-smart-contract-2510.21401
