Research
A Risk-Stratified Benchmark Dataset for Bad Randomness (SWC-120) Vulnerabilities in Ethereum Smart Contracts
Ethereum contracts that need random numbers — lotteries, games, token distributions — often derive them from values like block.timestamp or blockhash, which miners can predict or influence.

- arXiv
- 2601.09836
- Published
- 2026-01-14
- Authors
- Hadis Rezaei, Rahim Taheri, Francesco Palmieri
AI summary
A Risk-Stratified Benchmark Dataset for Bad Randomness (SWC-120) Vulnerabilities in Ethereum Smart ContractsOverview
Research area: Smart contract security — specifically the detection and benchmarking of Bad Randomness (Weak Sources of Randomness, SWC-120) vulnerabilities in Ethereum Solidity contracts.
Technical level: Intermediate. The paper is readable without deep formal methods background, but familiarity with Ethereum concepts (block attributes, modifiers, miners, oracles) helps considerably.
Scope (one sentence): The authors build and validate a large, risk-stratified, function-level-verified benchmark dataset of Ethereum contracts containing bad randomness vulnerabilities, and use it to measure how badly existing detectors perform.
What This Paper Is About
Ethereum contracts that need random numbers — lotteries, games, token distributions — often derive them from values like block.timestamp or blockhash, which miners can predict or influence. Existing detection tools and the small datasets used to evaluate them are weak: available benchmarks contain as few as 2 to 384 contracts, and they label a contract "protected" merely because a modifier like onlyOwner exists somewhere in the code, without checking it actually guards the vulnerable function. The paper's goal is to build a much larger, correctly labeled, risk-stratified benchmark that distinguishes a vulnerability's presence from its actual exploitability.
Key Contributions
- Largest validated SWC-120 benchmark dataset. From 47,398 SmartBugs-Wild contracts, keyword filtering yielded 17,466 candidates and the final dataset contains 1,752 vulnerable contracts plus 6 Safe ones (1,758 total in Table VI) — described as 51× larger than RNVulDet (34 vulnerable contracts), 5× larger than TaintSentinel (384), and 219× larger than SmartBugs Curated (8).
- Function-level mitigation validation. An algorithm checks whether a protective modifier is applied to the specific function containing the vulnerable pattern (and, for internal/private functions, to every public caller) rather than merely existing elsewhere in the contract.
- Four-level risk classification. Contracts are stratified as Safe, Low_Risk, Medium_Risk, and High_Risk based on which attacker types — external attackers, miners, or owners — can exploit them, replacing binary vulnerable/safe labels.
- Context-aware refinement. Keyword analysis on contract, function, and variable names separates legitimate uses of block attributes (mining/proof-of-work tokens, time tracking) from genuine randomness generation (lotteries, gambling).
- Empirical tool evaluation. Slither and Mythril are benchmarked against the new labels, exposing a coverage gap.
Main Findings
- Pattern matching on 17,466 filtered contracts: 1,903 contracts (10.8%) matched one or more of 58 regular expressions across 9 semantic groups (G1–G9).
- Dominant vulnerability patterns: G1 (direct modulo with block attributes) appeared in 687 contracts (36.1%), G2 (type cast from
keccak256/sha3touint) in 412 (21.7%), and G6 (seed/random variable from a predictable source) in 298 (15.7%). G1 and G2 together account for 57.8% of patterns; G4 and G9 each appeared in fewer than 50 contracts. - Labeler validated against ground truth: On a 32-contract ground-truth set compiled from the SWC Registry, SmartBugs Curated, and TaintSentinel, the labeler scored 22 true positives, 10 true negatives, 0 false positives, 0 false negatives — 100% accuracy, precision, recall, and F1.
- Contract-level protection labels are badly wrong: Of the 1,167 contracts initially classified Low_Risk, 572 (49.0%) were reclassified as false positives because
onlyOwnerdid not guard the vulnerable function. A second call-chain pass on the 335 that passed flagged 163 more (48.7%) where an internal vulnerable function was reachable through an unprotected public caller. Only 172 contracts (14.7% of the original 1,167) were confirmed Low_Risk — an 85.3% false positive rate in the initial classification. - Medium_Risk also corrected: Of the 47 initially classified Medium_Risk, 10 were false positives and 4 had no pattern inside any function body, leaving 37 confirmed.
- Context analysis rescued more labels: Of the 260 contracts where patterns sat outside function bodies, 138 (53.1%) were mining tokens and excluded, 108 (41.5%) were lottery applications labelled High_Risk, and 14 (5.4%) were flagged as Unknown; of those, 7 were excluded as time tracking and 7 were labelled High_Risk. In total 145 contracts were excluded as out of scope.
- Most vulnerable contracts have no protection at all: The final dataset is 87.8% High_Risk (1,543 contracts), 9.8% Low_Risk (172), 2.1% Medium_Risk (37), and 0.3% Safe (6).
- Existing tools detect essentially nothing: Slither (v0.11.3), run on all 1,752 vulnerable contracts, produced 0 true positives and 1,752 false negatives (accuracy 0.3%, recall 0%, F1 0%). Mythril (v0.24.8), run on a 76-contract sample (56 vulnerable, 20 safe), produced 0 true positives and 51 false negatives (accuracy 8.9%, recall 0%). Groups G2–G9, which the paper says account for 63.9% of the dataset, are entirely undetected by both tools; neither tool handles
keccak256hashing, type casting, or indirect variable usage.
Methodology in Plain English
The authors worked in five phases, starting from the 47,398 unique contracts in SmartBugs-Wild (collected from Etherscan):
- Keyword filtering. Keep only contracts mentioning at least one of
block.timestamp,blockhash,block.difficulty,block.number,block.coinbase, orblock.gaslimit— dropping the set to 17,466 contracts (a 63.1% reduction). They deliberately searched all contract domains, not just lottery/gaming as their earlier TaintSentinel work did. - Pattern labeling. Check each contract against 58 regular expressions in 9 groups covering direct modulo arithmetic,
keccak256/sha3casting, hashed values used with modulo, deprecatedblock.blockhashsyntax, stored block numbers, seed/random variable assignments, winner selection, and contextual keyword patterns. Contracts using Chainlink VRF are labelled Safe first. - Risk classification. Assign one of four levels based on who can exploit the contract: Safe (Chainlink VRF or a commit-reveal pair), Low_Risk (access control such as
onlyOwner), Medium_Risk (tx.origin == msg.senderchecks or future-block patterns), and High_Risk (no mitigation). - Function-level validation. Parse the source with bracket counting to extract every function, find which functions actually contain a bad-randomness pattern, and confirm the expected mitigation is attached to that function — or, for internal/private functions, to every public function that calls it. The algorithm returns CORRECT, FALSE_POSITIVE, or NO_PATTERN_IN_FUNCTIONS.
- Context refinement. For patterns found outside callable functions, inspect names for mining vocabulary (
mint,difficulty,nonce,mining,reward) versus lottery vocabulary (lottery,jackpot,prize,bet,gamble) and exclude or relabel accordingly.
Why This Matters
Impact on research. Small or mislabeled benchmarks propagate errors: tools trained on data that says "protected" whenever a modifier appears learn the wrong detection rule. This paper quantifies that error (49% of apparently protected contracts were exploitable) and supplies a dataset and validation scripts — publicly released on GitHub under CC BY 4.0 — that other researchers can build on. It also reframes the problem from binary present/absent to graded exploitability, arguing that remediation should target the 87.8% High_Risk contracts first.
Real-world applications:
- Lotteries and on-chain gambling, where predictable randomness lets an attacker or miner steer the winner.
- DeFi protocols, where randomness in liquidation or selection mechanisms can be gamed for profit (a domain the authors say TaintSentinel missed by restricting its search).
- Token distributions and airdrops, where predictable ordering or selection can be front-run.
- Security auditing and CI pipelines, where a benchmark with risk tiers lets teams prioritize which findings actually matter.
Industry relevance. The paper reports that 87.8% of contracts with bad randomness patterns have no mitigation at all, despite years of published research and tooling. Against OWASP's figure of over $1.42 billion lost across 149 documented incidents in 2024, and the SmartBillions attack that stole over 400 ether, the authors frame this as a gap between academic knowledge and developer practice — and as a direct argument that tool vendors need function-level analysis rather than contract-level pattern matching.
Future Directions
- Bytecode-only contracts. The current method requires available source code, so deployed contracts without verified source are excluded; extending analysis to bytecode is an open problem.
- Inter-contract call tracing. The authors do not follow randomness usage across contract boundaries, so vulnerabilities reached through external calls can be missed.
- Broader pattern coverage. The 58 regular expressions may miss implementations, and new patterns (the paper notes
block.prevrandaoandgasleft()were added ad hoc) will emerge as Solidity evolves. - Full-scale baseline evaluation. Mythril was only run on a 76-contract sample for computational reasons; a complete run, and evaluation of a wider set of tools, would test whether the 0% recall result generalizes.
- Transferring the validation method. The authors propose function-level mitigation validation as a template for improving label accuracy in other SWC vulnerability categories.
Target Audience
Smart contract security researchers and tool developers will get the most from this paper, particularly those building or benchmarking vulnerability detectors and those curating ML training data for code analysis. It is also directly useful to auditors and security engineers who need a realistic, risk-prioritized corpus of bad randomness examples, and to graduate students looking for a worked example of dataset construction methodology in program analysis. Readers need basic Solidity literacy; the algorithmic content is presented in accessible pseudocode.
Authors’ abstract
Many Ethereum smart contracts rely on block attributes such as block.timestamp or blockhash to generate random numbers for applications like lotteries and games. However, these values are predictable and miner-manipulable, creating the Bad Randomness vulnerability (SWC-120) that has led to real-world exploits. Current detection tools identify only simple patterns and fail to verify whether protective modifiers actually guard vulnerable code. A major obstacle to improving…
The full abstract on arXiv