Research journal · Security research
Evaluating Agent-Written Fuzz Tests
Can a coding agent do the harness-writing work of manually guided fuzzing on a real protocol? Measured on two DeFi protocols with known, human-triaged bugs.
In manually guided fuzzing, auditors write the harness: the flows, the preconditions, the state model and the invariants of one specific protocol. That is the expensive part, and the research question is whether a coding agent can do it. This note measures that on two real protocols from scfuzzbench, whose known bugs, found and triaged by humans, give the measure: known recall, the share of those bugs the agent’s tests reproduce. The benchmark’s own harnesses, counterexamples and bug lists were withheld from the agent.
The first experiment ran on Origin Dollar (OUSD), a rebasing stablecoin: six runs together reproduced 6 of the 12 known bugs, the same six that every hand-written harness in the benchmark reproduces; the best single run reproduced four. The second ran on Drips, a streaming-payments protocol: two runs of the current workflow spent 178 hours without reaching the state its one known bug needs. These are the results presented at DeFi Security Summit 2026 in Mumbai on 1 November.
The instrument: two protocols with known bugs
The earlier measurement of agent-written tests, the Daedaluzz benchmark, used synthetic mazes with the assertions already in the code. That answers how well an agent can reach a check that exists. It does not answer whether an agent can invent the actions and checks that a real protocol needs, and a real protocol has no complete answer key. A benchmark with known, human-triaged bugs is the closest fixed instrument: it does not measure everything the test finds, but every known bug it reproduces is a scenario and a property the agent wrote without seeing them.
scfuzzbench is a public benchmark in which Echidna, Medusa, Foundry and Recon run hand-written property harnesses against each protocol at one commit, each with a list of known bugs: real bugs that humans found and triaged in the protocols’ own fuzzing suites. Two targets were used:
- Origin Dollar (OUSD) at scfuzzbench commit
299ec6b6: a rebasing stablecoin with 12 known bugs, each marked as a known issue on one property of the protocol’s own Echidna suite. The list counts properties, not root causes, so twelve entries need not be twelve independent defects. - Drips at commit
c50d160e: a streaming-payments protocol with one known bug, D1, a real vulnerability found by an earlier fuzzing campaign and fixed upstream, which the benchmark reintroduces on purpose.
The agent’s checkout had the benchmark harnesses, recorded counterexamples, known-bug files and revealing comments removed. Both targets forked Ethereum mainnet at block 25,948,100. After each run, the agent’s transcripts were searched for reads of the excluded suites, property names or counterexamples; none occurred. Training familiarity with public targets cannot be excluded this way.
The workflow
The workflow, ai-fuzz, runs terminal coding agents (Codex CLI in these runs) as the steps of a loop on top of Wake, which compiles the project, generates Python types for every contract and runs the campaigns. Four roles, each with its own persistent session:
- Developer writes the test: flows (state-changing calls with fuzzed inputs) and invariants checked after each flow. It picks each campaign’s time cap, debugs a failing campaign in the paused campaign’s debugger, and either fixes the test or reports an issue.
- Reviewer runs when the developer’s queue is empty. It reads the diff since its last review and the line coverage, then lists tasks for missing flows, weak invariants and uncovered code. A review that lists nothing ends the run.
- Researcher answers what the code is supposed to do, so the test can check what it actually does. It never sees the implementation: it works in a temporary directory with a copy of the documentation and web search, excluding audit reports and vulnerability disclosures. Every answer states its confidence and cites its source.
- Triage gives every reported issue a verdict with evidence: true positive, false positive, duplicate or inconclusive. It also takes over a failure that twenty debugging sessions in a row did not resolve.
The audited Solidity is read-only for every role. Three test writers were used: GPT-5.6 Sol, Daybreak Blue, and GPT-6 Sol on the current version of the workflow. Two older versions preceded it; the triage role and the hand-over rule were added after the first OUSD run spent 1,124 debugging sessions on one assertion. Model and workflow version changed together, so the runs do not isolate a model effect.
Scoring: a first pass had a model read the known bugs next to the issues the run reported and propose matches. Every match was then read by hand and counts only when the same property fails by the same mechanism. Doubtful matches are not counted.
Results on OUSD
Six full runs across the three workflow versions reproduced 6 of the 12 known bugs between them, numbered K1 to K12 in the table below: K1, K2, K4, K7, K11 and K12. That is the same set each of the four baseline fuzzers reproduces with the benchmark’s hand-written harness, and none of the six that no tool reproduces. No single run reached the set: the best, run 7 with GPT-6 Sol, reproduced four in 6.3 hours and ended when the model provider refused to continue. Each run reproduced a different subset.
| Bug | Property broken | Agent runs | Echidna | Medusa | Recon | Foundry |
|---|---|---|---|---|---|---|
| K1 | changeSupply does not exactly reach the requested total supply | 1, 2, 5, 6 | 7–8 s | 3 s | <1 s | ≤25 min |
| K2 | Total supply can fall below the sum of balances | 5, 7, 8 | 2.6–20 min | 45–57 s | 17–30 s | ≤25 min |
| K3 | Non-rebasing supply can exceed total supply | none | none | none | none | none |
| K4 | Rebasing credits per token can increase | 7 | 8–23 s | 9–18 s | 3–15 s | ≤25 min |
| K5 | Non-rebasing supply can fall below non-rebasing balances | none | none | none | none | none |
| K6 | Minting can raise a balance by less than the amount | none | none | none | none | none |
| K7 | Burning can lower a balance by less than the amount | 7, 8 | 8–12 s | 12–15 s | <1–8 s | ≤25 min |
| K8 | Burning a full balance can leave a nonzero balance | none | none | none | none | none |
| K9 | Burning a full balance can revert | none | none | none | none | none |
| K10 | A recipient can receive less than the amount | none | none | none | none | none |
| K11 | A sender’s balance can drop by less than the amount | 7, 8 | 8–19 s | 12–24 s | <1–13 s | ≤25 min |
| K12 | A transfer within balance can revert | 2 | 7–11 min | 42–51 s | 19–40 s | ≤3.7 h |
Fuzzer columns are time to the first failure across scfuzzbench’s own runs on the same commit, with a harness someone had already written. The agent’s hours include writing and debugging that harness.
| Run | Test writer | Elapsed | Issues reported | Known bugs reproduced | Ended |
|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol | 24.0 h | 3 | K1 | stopped at 24 h; 1,124 debugging sessions on one assertion |
| 2 | Daybreak Blue | 24.0 h | 21 | K1, K12 | stopped at 24 h |
| 5 | Daybreak Blue | 29.2 h | 17 | K1, K2 | stopped after 13.8 h with no new issue |
| 6 | Daybreak Blue | 17.1 h | 12 | K1 | ended on its own, 113 reviewer tasks completed |
| 7 | GPT-6 Sol | 6.3 h | 23 | K2, K4, K7, K11 | model provider refused to continue |
| 8 | GPT-6 Sol | 17.1 h | 39 | K2, K7, K11 | model provider refused to continue |
Three further GPT-5.6 Sol starts were refused in the first step and are not in the table. Runs 7 and 8 are the only ones on the current workflow version, and the only ones that ended in a refusal.
Results on Drips
D1: a recipient’s claimable balance can lose funds that were already streamed, because a claim does not cap a stream’s start at the time its configuration was updated. The four baseline fuzzers trigger it on the same commit in 1.7 seconds (Recon), 15 seconds (Echidna), 21 seconds (Medusa) and 1.7 hours (Foundry).
| Run | Test writer | Workflow version | Elapsed | D1 | Ended |
|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol | older | 28.9 h | found at about 5.1 h | server reboot |
| 2 | Daybreak Blue | older | 28.8 h | found by 5.8 h | server reboot |
| 3 | GPT-6 Sol | current | 94.8 h | missed | stopped; 68 reviewer tasks open and growing |
| 4 | GPT-6 Sol | current | 83.6 h | missed | ended on its own, 111 reviewer tasks completed |
Missing test scenario, in both current-workflow tests: claim earned funds after the payment has started. The two misses share this one cause, visible in the tests themselves: neither test ever claimed from a stream whose start time had already passed. Every flow configured payments and claimed in the future-only case, so the accounting state D1 needs was never reached. Run 4 ended normally, with the reviewer satisfied, after 83.6 hours. Run 2 encountered the property names in a documentation listing about 20 hours after it had already reported D1.
Cost by test writer
| Test writer | OUSD | Drips | Refused starts | Cost | Cost per hour |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 1/12, 1 run | 1/1, 1 run | 3 on OUSD | $749.74 | $13.8–14.4 |
| Daybreak Blue | 3/12, 3 runs | 1/1, 1 run | none | $1,033.90 | $9.7–13.1 |
| GPT-6 Sol | 4/12, 2 runs | 0/1, 2 runs | none | $667.98 | $2.7–5.6 |
Scores are combined across each writer’s runs. Costs are estimated at list price from the agent transcripts, not invoices, and exclude infrastructure. In total: 13 starts, 354 hours, $2,451.62.
Conclusion
Is it useful? Yes. Without seeing the hand-written harnesses, the agent-written tests reached the same known recall on OUSD as every hand-written harness: the same six bugs.
What it is good for: writing the harness, the expensive part of manually guided fuzzing. A protocol without a fuzzing suite can get one: actions, a state model and checks, written as Wake tests that an auditor can read and rerun.
The limits: the agent chooses the scenarios, and it can miss the one that matters. On Drips it delivered a finished, passing test that never claimed earned funds after the payment had started, so it could not reach the known bug, and nothing in its own workflow noticed. It is also slow and uneven: hours of writing and debugging per run, at most four of the twelve OUSD bugs in any single run, and none of the bugs that no hand-written harness finds either.
So the auditor’s work moves from writing fuzz tests to reading them, and checking that their scenarios reach the states where funds can go wrong.
Vocabulary
Terms used in this article.
-
Protocol
A blockchain protocol is a defined set of rules and software interfaces that participants follow to reach consensus or provide an onchain application service.
-
Echidna
Echidna is an open-source Ethereum smart contract fuzzer that generates call sequences to falsify user-defined properties and Solidity assertions.
-
Fuzz testing
Automated testing that explores many generated inputs or action sequences to find violations of expected properties.
-
Invariant
A property that must remain true before and after every relevant state transition in a system.
-
Wake
An open-source Python framework for testing, fuzzing, and statically analysing Solidity, developed by ack3; tests and fuzz harnesses are Python.
-
Auditor
A smart contract auditor is a security researcher who reviews blockchain systems for exploitable behavior, unsafe assumptions, and implementation errors.