Why AI-Assisted Smart Contract Testing Belongs in Python

The implementation language of a smart contract is fixed by its deployment target. The language of its test suite is not. That distinction matters when an AI assistant helps scaffold fuzz tests, propose invariants, or build a mainnet-fork setup: the test language determines which region of the model’s competence the work exercises.

The available evidence does not show that models are universally best at Python, nor that writing tests in Python makes their logic correct. It shows a narrower and more useful result: code models are generally stronger in well-represented languages, Solidity remains low-resource, and Python is one of the most represented and most frequently evaluated programming languages in model training and benchmarking.

The training-data gap is measurable

The StarCoder corpus contains 60.40 GB of Python across 12,866,649 files and 0.85 GB of Solidity across 153,194 files. That is a volume difference of approximately 71 to 1.

Python is also among the largest language components of The Stack and The Stack v2. Solidity does not appear among the 32 languages for which The Stack v2 report publishes per-language statistics.

Training volume is not a perfect predictor. JavaScript or TypeScript can outperform Python for some models and tasks, and high-resource languages do not rank in a fixed order. The robust distinction is between high-resource languages with abundant examples and low-resource languages with sparse representation.

Scarcity affects code-generation performance

MultiPL-E translated standard code-generation problems into more than 18 programming languages and evaluated the same models on equivalent tasks. StarCoderBase reached 30.4 pass@1 on Python but approximately 10–12 on low-resource languages including R, D, and Racket. Across the evaluated models, performance was reliably lower for less frequent languages.

That result is older than today’s frontier models, but more recent research finds the same default behavior. CangjieBench evaluates models on a contamination-resistant benchmark for the low-resource Cangjie language. Direct generation performs poorly, while syntax constraints, retrieval, and agentic compiler feedback recover substantial capability.

That recovery is important. It suggests the underlying reasoning ability is not simply absent; the model lacks language-specific knowledge and feedback. Tooling can close much of the gap, but the gap exists by default.

Solidity remains difficult for current models

Current smart-contract benchmarks measure several different abilities, so their numbers should not be compared as though they were one leaderboard.

EVMbench contains 117 vulnerabilities from 40 audited repositories and measures detection, patching, and exploitation in executable environments. Its strongest evaluated configuration detected fewer than half of the vulnerabilities. A subsequent critique found that rankings and outcomes shift with scaffolding and evaluation setup, reinforcing the need to treat any single score as a measurement of a system, not a permanent capability ceiling.

Other 2026 studies report similarly mixed results:

• EvoVuln reports 52% macro-F1 for zero-shot GPT-5.2 on real-world Solidity vulnerability detection.
• A study of fifteen 2025–2026 models accepted at IEEE COMPSAC 2026 found balanced accuracy between 0.53 and 0.72 and measurable degradation under identifier renaming. Its conclusion is appropriately conservative: LLMs are not reliable standalone security auditors.
• PRIMG, using a small open model on three audited Code4rena projects, found only 2–4.6% of single-shot generated Solidity unit tests were correct. This is a limited experiment, but it remains one of the few published measurements aimed directly at Solidity test generation.

The evaluation ecosystem is also uneven. HumanEval and MBPP are Python-only. SWE-bench Multilingual and the Aider polyglot benchmark cover several languages, but neither includes Solidity. Models receive far more training signal and far more benchmark feedback for mainstream languages.

The closest comparison is suggestive, not definitive

Cross-benchmark comparisons can exaggerate the effect because a repository-level Solidity task is harder than a function-level Python exercise. SolEval, for example, should not be compared directly with Python HumanEval.

SolContractEval offers a closer, though still imperfect, comparison. The same model reached 12.1% on contract-level Solidity tasks and 29.6% on comparable Python class-level tasks. The paper’s authors describe the result as suggestive rather than rigorous, which is the right interpretation.

No published study yet gives us the ideal experiment: current frontier models generating matched Solidity and Python test suites for identical smart contracts, graded for compilation, behavioral coverage, and bug-finding ability.

Why the test language is a security decision

The implementation language of a smart contract is fixed by its deployment target. The language of its test suite is not.

When an assistant generates a Solidity test, it must simultaneously handle an underrepresented language, contract-specific syntax, framework conventions, and the security logic being tested. Moving the test harness to Python removes one avoidable source of model error. The smart contract still executes with the same EVM semantics; only the harness is written in a language where models have more training data and where developers have a mature debugging ecosystem.

This is one reason Wake uses Python for testing and fuzz testing Solidity contracts. Python is useful for:

• expressing state models with ordinary classes, dictionaries, and numeric code;
• generating structured inputs without compressing everything into Solidity syntax;
• using mature debuggers, profilers, data libraries, and test tooling;
• separating the model of expected behavior from the contract implementation; and
• reviewing AI-generated scaffolding in a language models handle more reliably.

The language choice improves the conditions under which the assistant works. It does not validate the result.

Human review still owns the security properties

AI is well suited to repetitive structure: imports, account selection, transaction wrappers, event-filtering loops, and basic state containers. Humans still need to verify the parts that determine whether a test can find a bug:

1. The property. Does the assertion or invariant express something that must actually hold?
2. The expected value. Was it derived independently before the transaction, or copied from the contract after execution?
3. The reached path. Did the generated test exercise the branch it claims to cover?
4. The failure model. Are all expected reverts, events, rounding rules, fees, and boundary cases accounted for?

An assistant can produce a beautiful test that only repeats the implementation. If the Python model contains the same branches and assumptions as the Solidity code, both can be wrong in the same way.

The practical conclusion

The evidence supports a disciplined claim, not a slogan:

• high-resource programming languages generally receive better default model performance;
• Solidity is underrepresented in both training corpora and evaluation suites;
• targeted fine-tuning, retrieval, compiler feedback, and agent scaffolding can narrow the gap; and
• using Python for the test harness places AI-assisted work in a better-supported language while preserving Solidity execution semantics.

Contracts remain in Solidity. Tests and fuzzing move to Python. The assistant handles more of the mechanical work in a language it knows better, and the reviewer spends more attention on the part that determines security: the meaning of the test.

References

• Li et al., StarCoder: May the Source Be with You!, 2023.
• Kocetkov et al., The Stack, 2022.
• Lozhkov et al., StarCoder 2 and The Stack v2, 2024.
• Cassano et al., MultiPL-E, IEEE TSE, 2023.
• Cheng et al., CangjieBench, 2026.
• Wang et al., EVMbench, 2026.
• Bouafif et al., PRIMG, EASE 2025.
• Peng et al., SolEval, EMNLP 2025.
• Ye et al., SolContractEval, 2025.