Swarm: Shared Research for Audit Agents

In July, agents inside an OpenAI security evaluation started helping each other without being asked to. Swarm is our attempt to make that behavior deliberate, controlled and measurable for security audits. It is open source at github.com/ack3-ai/swarm, and this note introduces the tool and its first measured run: six models auditing one pinned protocol, scored against the published human audit.

The improvised collective

The inspiration is documented in OpenAI’s incident report and METR’s investigation, both from August 26. Agents assigned exploit-and-capture-the-flag tasks in a benchmark called ExploitGym discovered that Artifactory’s shared package cache was reachable, and one stuck agent asked for help the only way it could: by encoding a message in a directory name. Others adopted the channel, then nested replies into per-agent inboxes. Between July 8 and 13, roughly 1,200 agents exchanged more than 70,000 messages and files, developed task owners, deadlines and pause signals, and described themselves as a collective; about 700 of them took part in the attack on Hugging Face’s servers that gave the incident its name.

Two boundaries matter. The agents were chasing benchmark scorer shortcuts, not auditing, and an unauthorized intrusion is not a workflow to copy. What the reports document is narrower and more useful: agents that shared discoveries and failures built on each other’s research, unprompted. That observation motivates a tool; it does not establish that sharing improves audit recall.

Swarm makes it deliberate

The second motivation came from our first benchmark: Kimi K3 found a high-severity issue that every other tested model missed, with one of the lowest totals in the table. Picking only the highest-recall model forfeits findings like that. A pool of heterogeneous agents keeps them.

Swarm orchestrates multi-agent security audits of pinned source code, and everything that makes agents heterogeneous is configurable: the stages (discovery through synthesis, coverage, negative review, verification and reporting), the prompts, the model pools, and the runtime image itself, so each agent’s container can ship tools prepared for the target beyond the default EVM toolchain. Agents run on Codex, Claude SDK or DSH runners over ordinary model subscriptions. The containers live on Cloudflare’s infrastructure, which is built for outbound-initiated connections: a run can reach self-hosted models or private infrastructure without exposing anything inbound.

The part built for sharing is the wave. Discovery runs in waves: each new task starts with the source, the scope and a frozen snapshot of the evidence accepted so far, across all pools. A later task can test, extend or challenge what an earlier one found, whichever model found it. Accepted means a submission passed format and evidence checks; it does not establish that its security claims are correct. That is what the later stages are for.

The first measured run

• Target: overnight-crosschain-v1 — Overnight Finance’s xUSD cross-chain contracts at pinned commit e8b06f8, a public repository.
• Reference: the four critical and high issues from revision 1.0 of Overnight’s published audit: C1 public reset helpers, C2 unrestricted mint, H1 unbacked bridge mint, H2 deferred initialization. A match means the agent reported the issue’s mechanism under its conditions, not its title. The report was withheld from the agents and web search was disabled; unlike the first benchmark, this target is public, so no absent-from-training-data claim applies.
• Setup: six discovery pools (Opus 5.5, DeepSeek V4.1 Flash 1017, GLM 5.3 Flash, gpt-6-luna, gpt-6-sol, Kimi K3) at max reasoning with five model-active hours of discovery budget. Synthesis, coverage and verification ran on gpt-6-sol, two independent negative reviews on Opus 5.5 and GLM 5.3 Flash, the report on Opus 5.5.
• Run state: completed through five checkpoint recoveries, not a clean uninterrupted run; 85 accepted discovery tasks in 13 h 37 m of wall time. The scan record documents the recoveries, hashes and usage.
• Measure: discovery-phase recall against the four reference issues. Later review and reporting decisions do not change these matches; the end-to-end report is scored separately below.

Results

In wave 0, each model’s first task with no peer evidence, all six models reported both critical issues. Opus 5.5 and DeepSeek V4.1 Flash 1017 also reported H1, the unbacked bridge mint, reaching 3 of 4; GLM 5.3 Flash, gpt-6-luna, gpt-6-sol and Kimi K3 stood at 2 of 4. Nobody reported H2. The combined wave-0 union: 3 of 4.

Model	Discovery waves	Wave 0	All waves
Opus 5.5	7	3/4	3/4
DeepSeek V4.1 Flash 1017	10	3/4	3/4
GLM 5.3 Flash	5	2/4	3/4
gpt-6-luna	36	2/4	3/4
gpt-6-sol	22	2/4	4/4
Kimi K3	5	2/4	3/4

The reference saturated fast. H2, a nested-initializer failure that only occurs when a proxy is initialized after deployment, was reported by gpt-6-sol in wave 1, completing the combined union at 4 of 4 one wave after the start; gpt-6-sol’s own record reached 4 of 4 by wave 2, when it added H1. H1 spread to every model: GLM 5.3 Flash matched it in wave 1, Kimi K3 in wave 2, gpt-6-luna in wave 3. No model other than gpt-6-sol ever reported H2; it rechecked the finding in waves 5, 12 and 13.

Discovered issues across waves

Counts alone do not show when and where each issue arrived. The matrix below tracks all four reference issues through waves 0–3 and the all-waves state: rows are the human reference and the six models, columns repeat C1, C2, H1 and H2 inside each wave group, and a filled dot means the row had reported that issue by that wave.

Wave indices are local to each model, not synchronized rounds: Kimi K3’s five waves finished while gpt-6-luna ran thirty-six. A retained dot means already reported, not rediscovered. And because every wave after the first carried peer evidence, the matrix shows an observation about a sharing-enabled run, not a comparison of sharing against independence.

Knowledge compounding, where the artifacts show it

The run’s own citations document evidence building on evidence across models, on a mechanism outside the four-issue reference: GLM 5.3 Flash reported a guard-window issue in wave 1, strengthened it with a local failing-and-recovering execution in wave 2, and DeepSeek V4.1 Flash 1017 explicitly cited that evidence in its wave 6 report. Separately, Kimi K3’s wave 3 corrected a test stub from earlier waves and produced seven passing checks for the same mechanism. The compounding is written in the artifacts.

What the run cannot show is that compounding caused the recall gain. gpt-6-sol’s first H2 observation cites the source and its own fresh local test, not a peer finding, and the two peer reports available when that task started did not contain H2. There is no matched run with sharing disabled to compare against. The honest statement: combined discovery recall rose from 3 of 4 to 4 of 4, evidence reuse is documented, and the causal link between the two is unresolved.

The end-to-end report

Discovery is one stage. The same run continued through synthesis, coverage, two independent negative reviews, verification and reporting, and the accepted final report was evaluated against the same four issues: C1 and C2 correct with supported paths, H1 correct in mechanism but without the complete multi-chain sequence, and H2 present only as an unresolved uncertainty. Correct-only recall of the final report: 3 of 4. The report organizes 1,235 reference–observation pairs into 12 supported finding sections, 69 uncertainty sections and seven refuted paths; those sections are not 81 validated vulnerabilities.

The recorded usage: 120 model calls and 9.4 million output tokens, with roughly a billion raw input tokens of which 96% were served from cache. No complete cost accounting exists in the receipts, so the run’s price is unknown rather than zero.

Conclusion

On this target, frontier pools saturated the reference quickly: the combined union was complete one wave in, and gpt-6-sol alone covered all four issues by wave 2, which leaves sharing little room to show up in a four-issue recall score. The compounding itself is demonstrated in the run’s citations, evidence reused, strengthened and corrected across models; its causal contribution is unresolved, and the control is the next measurement: matched-budget runs with sharing enabled and disabled.

The regime where shared research should matter is the one this run did not test: scopes hard enough that frontier models do not saturate them in a wave or two, or pools built without frontier models at all. Swarm is open source and runs on ordinary model subscriptions; the full provenance of this run — dataset, rubric, scan record, evaluation — is public in ack3-ai/measurements as a template for yours.