Research journal · Security research

Evaluating Agent-Written Fuzz Tests

Can a coding agent do the harness-writing work of manually guided fuzzing on a real protocol? Measured on two DeFi protocols with known, human-triaged bugs.

research/agent-written-fuzz-tests-on-two-protocols figure / technical visual
Matrix of twelve known Origin Dollar bugs against ten test writers. Echidna, Medusa, Recon and Foundry with the benchmark's hand-written harness each reproduce the same six: K1, K2, K4, K7, K11 and K12. Six agent-written test runs reproduce one to four each; combined they reproduce the same six. No writer reproduces K3, K5, K6, K8, K9 or K10.
Which of the twelve known OUSD bugs each test writer reproduced. Filled dot: reproduced.

In manually guided fuzzing, auditors write the harness: the flows, the preconditions, the state model and the invariants of one specific protocol. That is the expensive part, and the research question is whether a coding agent can do it. This note measures that on two real protocols from scfuzzbench, whose known bugs, found and triaged by humans, give the measure: known recall, the share of those bugs the agent’s tests reproduce. The benchmark’s own harnesses, counterexamples and bug lists were withheld from the agent.

The first experiment ran on Origin Dollar (OUSD), a rebasing stablecoin: six runs together reproduced 6 of the 12 known bugs, the same six that every hand-written harness in the benchmark reproduces; the best single run reproduced four. The second ran on Drips, a streaming-payments protocol: two runs of the current workflow spent 178 hours without reaching the state its one known bug needs. These are the results presented at DeFi Security Summit 2026 in Mumbai on 1 November.

The instrument: two protocols with known bugs

The earlier measurement of agent-written tests, the Daedaluzz benchmark, used synthetic mazes with the assertions already in the code. That answers how well an agent can reach a check that exists. It does not answer whether an agent can invent the actions and checks that a real protocol needs, and a real protocol has no complete answer key. A benchmark with known, human-triaged bugs is the closest fixed instrument: it does not measure everything the test finds, but every known bug it reproduces is a scenario and a property the agent wrote without seeing them.

scfuzzbench is a public benchmark in which Echidna, Medusa, Foundry and Recon run hand-written property harnesses against each protocol at one commit, each with a list of known bugs: real bugs that humans found and triaged in the protocols’ own fuzzing suites. Two targets were used:

  • Origin Dollar (OUSD) at scfuzzbench commit 299ec6b6: a rebasing stablecoin with 12 known bugs, each marked as a known issue on one property of the protocol’s own Echidna suite. The list counts properties, not root causes, so twelve entries need not be twelve independent defects.
  • Drips at commit c50d160e: a streaming-payments protocol with one known bug, D1, a real vulnerability found by an earlier fuzzing campaign and fixed upstream, which the benchmark reintroduces on purpose.

The agent’s checkout had the benchmark harnesses, recorded counterexamples, known-bug files and revealing comments removed. Both targets forked Ethereum mainnet at block 25,948,100. After each run, the agent’s transcripts were searched for reads of the excluded suites, property names or counterexamples; none occurred. Training familiarity with public targets cannot be excluded this way.

The workflow

The workflow, ai-fuzz, runs terminal coding agents (Codex CLI in these runs) as the steps of a loop on top of Wake, which compiles the project, generates Python types for every contract and runs the campaigns. Four roles, each with its own persistent session:

  • Developer writes the test: flows (state-changing calls with fuzzed inputs) and invariants checked after each flow. It picks each campaign’s time cap, debugs a failing campaign in the paused campaign’s debugger, and either fixes the test or reports an issue.
  • Reviewer runs when the developer’s queue is empty. It reads the diff since its last review and the line coverage, then lists tasks for missing flows, weak invariants and uncovered code. A review that lists nothing ends the run.
  • Researcher answers what the code is supposed to do, so the test can check what it actually does. It never sees the implementation: it works in a temporary directory with a copy of the documentation and web search, excluding audit reports and vulnerability disclosures. Every answer states its confidence and cites its source.
  • Triage gives every reported issue a verdict with evidence: true positive, false positive, duplicate or inconclusive. It also takes over a failure that twenty debugging sessions in a row did not resolve.

The audited Solidity is read-only for every role. Three test writers were used: GPT-5.6 Sol, Daybreak Blue, and GPT-6 Sol on the current version of the workflow. Two older versions preceded it; the triage role and the hand-over rule were added after the first OUSD run spent 1,124 debugging sessions on one assertion. Model and workflow version changed together, so the runs do not isolate a model effect.

Scoring: a first pass had a model read the known bugs next to the issues the run reported and propose matches. Every match was then read by hand and counts only when the same property fails by the same mechanism. Doubtful matches are not counted.

Results on OUSD

Six full runs across the three workflow versions reproduced 6 of the 12 known bugs between them, numbered K1 to K12 in the table below: K1, K2, K4, K7, K11 and K12. That is the same set each of the four baseline fuzzers reproduces with the benchmark’s hand-written harness, and none of the six that no tool reproduces. No single run reached the set: the best, run 7 with GPT-6 Sol, reproduced four in 6.3 hours and ended when the model provider refused to continue. Each run reproduced a different subset.

BugProperty brokenAgent runsEchidnaMedusaReconFoundry
K1changeSupply does not exactly reach the requested total supply1, 2, 5, 67–8 s3 s<1 s≤25 min
K2Total supply can fall below the sum of balances5, 7, 82.6–20 min45–57 s17–30 s≤25 min
K3Non-rebasing supply can exceed total supplynonenonenonenonenone
K4Rebasing credits per token can increase78–23 s9–18 s3–15 s≤25 min
K5Non-rebasing supply can fall below non-rebasing balancesnonenonenonenonenone
K6Minting can raise a balance by less than the amountnonenonenonenonenone
K7Burning can lower a balance by less than the amount7, 88–12 s12–15 s<1–8 s≤25 min
K8Burning a full balance can leave a nonzero balancenonenonenonenonenone
K9Burning a full balance can revertnonenonenonenonenone
K10A recipient can receive less than the amountnonenonenonenonenone
K11A sender’s balance can drop by less than the amount7, 88–19 s12–24 s<1–13 s≤25 min
K12A transfer within balance can revert27–11 min42–51 s19–40 s≤3.7 h

Fuzzer columns are time to the first failure across scfuzzbench’s own runs on the same commit, with a harness someone had already written. The agent’s hours include writing and debugging that harness.

RunTest writerElapsedIssues reportedKnown bugs reproducedEnded
1GPT-5.6 Sol24.0 h3K1stopped at 24 h; 1,124 debugging sessions on one assertion
2Daybreak Blue24.0 h21K1, K12stopped at 24 h
5Daybreak Blue29.2 h17K1, K2stopped after 13.8 h with no new issue
6Daybreak Blue17.1 h12K1ended on its own, 113 reviewer tasks completed
7GPT-6 Sol6.3 h23K2, K4, K7, K11model provider refused to continue
8GPT-6 Sol17.1 h39K2, K7, K11model provider refused to continue

Three further GPT-5.6 Sol starts were refused in the first step and are not in the table. Runs 7 and 8 are the only ones on the current workflow version, and the only ones that ended in a refusal.

Results on Drips

D1: a recipient’s claimable balance can lose funds that were already streamed, because a claim does not cap a stream’s start at the time its configuration was updated. The four baseline fuzzers trigger it on the same commit in 1.7 seconds (Recon), 15 seconds (Echidna), 21 seconds (Medusa) and 1.7 hours (Foundry).

RunTest writerWorkflow versionElapsedD1Ended
1GPT-5.6 Sololder28.9 hfound at about 5.1 hserver reboot
2Daybreak Blueolder28.8 hfound by 5.8 hserver reboot
3GPT-6 Solcurrent94.8 hmissedstopped; 68 reviewer tasks open and growing
4GPT-6 Solcurrent83.6 hmissedended on its own, 111 reviewer tasks completed

Missing test scenario, in both current-workflow tests: claim earned funds after the payment has started. The two misses share this one cause, visible in the tests themselves: neither test ever claimed from a stream whose start time had already passed. Every flow configured payments and claimed in the future-only case, so the accounting state D1 needs was never reached. Run 4 ended normally, with the reviewer satisfied, after 83.6 hours. Run 2 encountered the property names in a documentation listing about 20 hours after it had already reported D1.

Cost by test writer

Test writerOUSDDripsRefused startsCostCost per hour
GPT-5.6 Sol1/12, 1 run1/1, 1 run3 on OUSD$749.74$13.8–14.4
Daybreak Blue3/12, 3 runs1/1, 1 runnone$1,033.90$9.7–13.1
GPT-6 Sol4/12, 2 runs0/1, 2 runsnone$667.98$2.7–5.6

Scores are combined across each writer’s runs. Costs are estimated at list price from the agent transcripts, not invoices, and exclude infrastructure. In total: 13 starts, 354 hours, $2,451.62.

Conclusion

Is it useful? Yes. Without seeing the hand-written harnesses, the agent-written tests reached the same known recall on OUSD as every hand-written harness: the same six bugs.

What it is good for: writing the harness, the expensive part of manually guided fuzzing. A protocol without a fuzzing suite can get one: actions, a state model and checks, written as Wake tests that an auditor can read and rerun.

The limits: the agent chooses the scenarios, and it can miss the one that matters. On Drips it delivered a finished, passing test that never claimed earned funds after the payment had started, so it could not reach the known bug, and nothing in its own workflow noticed. It is also slow and uneven: hours of writing and debugging per run, at most four of the twelve OUSD bugs in any single run, and none of the bugs that no hand-written harness finds either.

So the auditor’s work moves from writing fuzz tests to reading them, and checking that their scenarios reach the states where funds can go wrong.

Contributor

About the author.

Vocabulary

Terms used in this article.