Benchmarking Frontier Models on Unpublished Audits

Nine frontier model configurations re-ran five unpublished audits through one harness. The strongest, Opus 5 at ultra reasoning, found 35 verified critical and high findings, compared with 32 in the revision-one human reports; the lowest, GLM 5.2 at max and the DeepSeek V4 Pro 0813 snapshot, reached 16.

Why unpublished audits

Every public benchmark eventually leaks into training data, and a model that has seen the answer key measures memory, not capability. The five engagements in this benchmark were delivered between 2022 and 2025 and were never published: not the code, not the reports. No model in the field has trained on either side of the ground truth.

Method

• Corpus: 5 unpublished audits, pinned to the commits the original engagements reviewed.
• Baseline: the 32 critical and high issues in the revision-one human reports. Revision one is a single report pass delivered inside a fixed engagement window, so the baseline is a snapshot of that pass, not a ceiling on what human auditors find.
• Ground truth: report-matched findings plus model-only findings subsequently validated by human reviewers.
• Setup: the same ack3 AI scan harness for every model at the stated reasoning level.
• Counting: unvalidated model candidates are excluded for every model.
• Measure: verified critical and high issues found: recall against the 40-issue verified union. We did not score false positives in this run.

Results

Two models cleared the human baseline: Opus 5 at ultra reasoning found 35 verified issues and GPT-5.6 Sol at ultra found 33. Opus 4.7 at max and GPT-5.6 Sol at xhigh stopped one issue short at 31, Grok 4.6 at xhigh reached 29, and the remaining counts fall from Opus 5 at xhigh’s 27 through Kimi K3 to GLM 5.2 and the DeepSeek V4 Pro 0813 snapshot at 16.

Later report revisions and four human-validated model-only findings extend the verified union beyond the 32-issue revision-one baseline to 40 issues, so a model can exceed the baseline while still missing verified issues.

Discoverer	Reasoning	Critical	High	Total
Human auditors, revision one	n/a	16	16	32
Opus 5	ultra	18	17	35
GPT-5.6 Sol	ultra	16	17	33
Opus 4.7	max	18	13	31
GPT-5.6 Sol	xhigh	16	15	31
Grok 4.6	xhigh	17	12	29
Opus 5	xhigh	17	10	27
Kimi K3	max	14	10	24
GLM 5.2	max	11	5	16
DeepSeek V4 Pro 0813	max	9	7	16

Discovered issues

Counts alone do not say whether the models found the same issues. The matrix below breaks the totals down to issue identity: each of the 40 columns is one anonymized verified issue, color encodes severity, and a filled dot means that row found it.

The matrix shows that Opus 5 at ultra’s 35 is not the human result plus three: 29 of its issues match the revision-one reports, six sit outside that revision in the wider union, and it missed three of the auditors’ 32, one critical and two high.

It also shows the models do not converge on one issue set. Only nine of the 40 issues were found by all nine models, and Opus 5 at ultra and GPT-5.6 Sol at ultra missed the same four issues. Human auditors had reported two of those in revision one, including one critical that Opus 4.7 at max also caught; one high was found by Kimi K3 at max alone; and the last was found by no model at all, appearing only in a later human report revision. The DeepSeek V4 Pro 0813 snapshot’s 16 all appear in other rows.

Conclusion

Opus 5 and GPT-5.6 Sol, both at ultra reasoning, cleared the revision-one human baseline on five unpublished audits, and discovery does not converge: even they missed verified issues that human auditors and cheaper models caught. The measure is deliberately narrow: five audits, critical and high severities, and recall only.

We extend the benchmark as new models ship, and this page tracks the current table.

Revision history

• 2026-08-17 — edition 2. Added Grok 4.6 at extra-high reasoning and replaced the DeepSeek V4 Pro run with its 0813 snapshot, both under the same ack3 AI scan harness. The verified-issue union, the human baseline, and every previously published model are unchanged; the newer DeepSeek snapshot rose from 11 to 16. Every figure now renders from a committed benchmark dataset, so each number on this page is reproducible.
• 2026-08-03 — edition 1. First publication: eight model configurations measured against five unpublished audits, with the severity table and the discovered-issues matrix.