Benchmarking Frontier Models on Unpublished Audits

The best frontier model recovered 30 of the 32 critical and high findings from five unpublished audits in the ack3 archive; the weakest recovered 11. We re-ran each audit through four frontier models in one identical harness and counted how many of the reports’ validated findings each model matched.

Why unpublished audits

Every public benchmark eventually leaks into training data, and a model that has seen the answer key measures memory, not capability. The five engagements in this benchmark were delivered between 2022 and 2025 and were never published: not the code, not the reports. No model in the field has trained on either side of the ground truth.

Method

• Corpus: 5 unpublished audits from the ack3 archive, pinned to the commits the original engagements reviewed.
• Ground truth: the validated findings of each engagement’s audit report.
• Setup: the same agentic harness for every model, each at its best available configuration (maximum reasoning).
• Counting: a model’s finding counts only when it matches a validated finding from the engagement. Candidate findings a model raised that we have not validated are excluded, for every model.
• Measure: critical and high findings recovered. This is recall against the human reports; we did not score false positives in this run.

Results

Across the five audits, the original reports contain 32 critical and high findings; the best model recovered 30, the weakest 11.

Discoverer	Critical	High	Total
Auditors (original reports)	16	16	32
Claude Opus 4.7	18	12	30
Kimi K3	14	9	23
GLM 5.2	11	5	16
DeepSeek V4 Pro	6	5	11

Critical and high are the report severities of the matched findings. A model can exceed the original reports in one severity column, as Claude Opus 4.7 does on criticals, by recovering findings validated after the original engagement; the totals still count only validated findings.

What this does and does not show

Two findings separate the best model from the human reports, and in security work that gap is not a rounding error: any one missed critical is an incident if it ships.

The spread matters more than the ceiling. Under identical conditions, on identical code, recall ranged from 30 to 11 of 32. A workflow that treats frontier models as interchangeable inherits the weakest number in that column.

The limits: five audits, critical and high severities, recall only. A model that raises many invalid candidates pays no penalty in this measure; precision is the next thing we want to quantify.

We extend the benchmark as models ship, and this page tracks the current table.