Benchmarking EVM Fuzzers Against an Agent-Written Harness

Three coverage-guided EVM fuzzers ran 45 eight-hour campaigns on the Daedaluzz maze benchmark and plateaued at 71 to 73 of 153 reachable assertions per tool; taken together, the three found 90. An agent-written harness, combining source parsing with SMT-derived inputs, reproduced all 153 in 79 minutes of a single campaign. The gap did not come from transaction throughput, and the measurements locate where it did come from.

These measurements were presented at Web3 Security Summit Belgrade on 2026-08-26.

Why Daedaluzz

Daedaluzz is a benchmark generator by ConsenSys Diligence. Each generated maze is a Solidity contract holding a position on a 7×7 grid that persists across transactions. Move functions take eight uint64 parameters, and cells hide assertions behind chains of arithmetic guards. The score is the number of distinct assertion violations, not line coverage and not transaction count, so every tool is graded by the same currency: bugs reached.

This measurement uses the L32 maze set: generated guard constants are capped at 2³²−1 while inputs remain uint64, arithmetic is unchecked, and the five mazes are generated from fixed seeds. Solving every guard with Z3 and replaying the solutions established that exactly 153 assertions are reachable across the five mazes; that count is the denominator everywhere below.

One cell from maze 0 shows the shape of the problem:

if (x == 0 && y == 5) {
  if (p6 != uint64(uint64(2398633357) + p1)) {
    if (p4 == uint64(p5 * p5)) {
      emit AssertionFailed("5");
      assert(false);
    }
  }
}

Reaching this assertion requires walking to cell (0,5) across multiple transactions and then satisfying arithmetic constraints over the inputs, including an exact relation between two 64-bit values. That combination, multi-transaction state plus constrained inputs, is what the benchmark isolates.

Method

• Tools: Echidna v2.3.3, Foundry v1.7.1 (commit 4072e487), and Wake at commit d33810e with the revm backend. All contracts compiled with solc 0.8.19, optimizer at 200 runs.
• Coverage campaigns: 3 seeds × 5 mazes × 3 tools = 45 runs of 8 hours each, one physical core per run. Reported endpoints are medians across seeds, summed over the five mazes.
• Throughput: measured in separate 20-minute-target runs, 15 tasks per tool, one core per task. Transaction counts come from each tool’s own output, and active fuzzing time excludes setup.
• Agent-written harness: a single campaign covering all five mazes in one process. An agent built the harness iteratively: navigation from parsed source (breadth-first search with safe transit inputs), guard inputs from Z3 with every parameter modeled as a 64-bit bitvector preserving unchecked wrap, and a simplify → bit-blast → solve fallback for queries the default solver returns unknown on. Every satisfying model is independently re-checked in Python, and every emitted reproduction is replayed against a fresh deployment. The replayed panic, not the solver result, is what counts toward the score.

Results: eight hours of traditional fuzzing

The three tools finish within two bugs of each other. Median endpoints after eight hours are Wake 73, Foundry 72, and Echidna 71 of 153 reachable assertions. Their findings overlap heavily rather than complementing one another: taken together, the three tools reached 90 of the 153, leaving 63 assertions that no traditional campaign in this measurement found. All three curves flatten early: most of each total arrives in the first hour, and the remaining budget adds a small tail.

Throughput is not the constraint

The tools differ enormously in raw speed and barely at all in outcome.

Engine	Transactions / s	vs Echidna	Distinct asserts / 1M tx
Foundry	30,186	11.6×	0.32
Wake	29,931	11.5×	0.37
Echidna	2,600	1.0×	3.36

An 11.6× spread in transaction rate produced a two-bug spread in eight-hour coverage. Per transaction, the ranking inverts: Echidna’s coverage-guided corpus extracts roughly ten times more distinct assertion failures per million transactions than the faster generators. The efficiency column is a 20-minute-window metric from the separate throughput runs, not an eight-hour ratio, and throughput is machine-specific; the direction of the inversion is the result, not the exact multiples.

The reading is that once the shallow assertions are exhausted, all three tools are limited by the same thing: the probability that generated inputs satisfy the remaining guards.

Where the probability mass sits

Wake’s unsigned integer generator makes that limit concrete. Five percent of draws pick one of five edge cases (0, 1, 2⁶³−1, 2⁶³, 2⁶⁴−1). The other ninety-five percent first pick a bit length from 1 to 64, then draw that many random bits. Because 32 of the 64 bit-length choices produce values under 2³², the generator places 50.98% of all draws below 2³². Uniform sampling over uint64 would place 2⁻³², about one draw in 4.3 billion, in the same range.

The same shape governs exact hits. One specific 32-bit constant has probability about 6.9×10⁻¹² under this prior against 5.4×10⁻²⁰ under uniform sampling, roughly eight orders of magnitude. For guard chains built from equality comparisons against constants, the generator prior, not engine speed, determines the hit rate. This is a general prior in Wake’s generators, not benchmark-specific code.

The agent-written harness

The agent-written harness reached 150 of the 153 assertions at minute 37 and completed the set at minute 79. It is a single campaign, so the comparison is valid for completion and wall time; variance across runs is unavailable at n=1.

What changed is where the search happens. The harness moved guard satisfaction out of random generation and into an SMT solver, and moved navigation out of random walks and into breadth-first search over the parsed maze. The runtime still deploys contracts, sends transactions, and checks assertions at scale; the discovery work moved into harness construction. That division, an executable model built by an agent and explored by the fuzzer, is the same structure as manually guided fuzzing with the manual part replaced.

Two scope notes bound the claim. The mazes are synthetic and deliberately favor exactly the kind of guard an SMT solver dispatches. And Daedaluzz is public, so familiarity with the benchmark family in the agent’s training data cannot be excluded, although the L32 constants are freshly generated.

What an agent-built model inherits

Replacing the manual part raises problems that the benchmark does not measure and that showed up in the surrounding workflow.

A model derived from the implementation can inherit its bugs. If a contract credits twice the deposited value by mistake and the agent translates that behavior into the differential model, the equality assertion between model and contract passes; the oracle fails exactly where the implementation does. An independent invariant, such as credited balances never exceeding assets held, detects what translation cannot. Whether an agent reliably derives such independent properties, rather than restating source behavior, is an open question.

Independent agent runs on the same task also diverge: different harnesses, different model depth, different assertions, and a different set of found issues. A harness that builds, runs, and passes says nothing by itself about whether the encoded security model is good enough, so a production workflow needs an explicit stopping rule and a quality bar for the harness, separate from task completion.

A datapoint from a real audit

One anonymized audit engagement provides a non-synthetic reference point. The full human report contained 7 critical, 3 high, and 4 medium issues. The manually guided fuzzing portion of that engagement, 17 person-days of auditor time, found 6 critical, 0 high, and 1 medium of them. Two independent agent runs of the same fuzzing task found 4 critical, 1 high, and 1 medium in 84 elapsed hours, and 5 critical, 3 high, and 2 medium in 28 elapsed hours.

The units differ: person-days measure effort while the agent numbers measure wall time, so the rows are context, not a controlled comparison. The run-to-run divergence is the controlled part of the observation, and it points the same direction as the benchmark: the longer agent run reported fewer findings in every listed severity than the shorter one.

Limitations

• The benchmark is synthetic: unchecked arithmetic, capped constants, and guard chains that reward SMT solving. No L32 campaign measured a real contract.
• The agent-written harness result is one campaign. Completion and wall time are measured; variance is not.
• Throughput and efficiency come from separate 20-minute-target runs on one pinned machine and do not transfer as absolute numbers.
• Daedaluzz is public; benchmark familiarity in the agent’s training data cannot be ruled out.
• The real-audit rows mix effort with elapsed time and cover a single engagement.

Conclusion

On this benchmark, the binding constraint of traditional fuzzing moved from execution speed to input construction: an 11.6× throughput spread bought two bugs, while replacing random input search with solver-derived inputs completed a suite that eight-hour campaigns left more than half unfinished. The agent did not replace the fuzzer; it replaced the part of the workflow that decides what the fuzzer should try. What remains open, and what we are measuring next, is variance across agent runs, stopping rules for harness construction, and whether agents can derive oracles independent of the implementation under test.