Static Analysis MCP Tools in an AI Audit Benchmark

Static analysis gives an AI model deterministic facts about code: the same query on the same code returns the same diagnostics, definitions, references, and call relationships. The model’s reasoning is non-deterministic. We tested whether that stable layer improved audit results.

Three GPT-5.6 models reviewed the same Go codebase with static-analysis MCP tools available and unavailable. Among the 134 cases that produced usable verdicts in both setups for all three models, recall improved for 0 of 3. Precision was higher for all three because the tools-on runs dismissed more false positives. LLM cost per benchmark run increased by 1.1% to 14.3%.

These measurements were presented in “Does static analysis actually help AI?” at ETHBelgrade 2026.

A deterministic helper for non-deterministic reasoning

Text search can locate the name withdraw. Static analysis can resolve which symbol it refers to, where that symbol is defined, which code references it, and which functions call it. On the same code, those results are reproducible.

That makes static analysis a deterministic helper inside a non-deterministic security review. It gives an AI model stable facts for navigation and verification. It does not make the model’s reasoning deterministic, and it does not answer the security question by itself.

The questions that matter in an audit sit one level higher:

can untrusted input reach this release path?
which state changes occur before this external call?
can an attacker-controlled implementation satisfy this interface?

The MCP did not answer those questions directly. It provided eleven Go-analysis functions that returned:

• workspace state, package listings, and diagnostics;
• symbols, definitions, references, and interface implementations;
• imports;
• direct callers and callees.

It had no taint-analysis query, vulnerability detector, consensus-specific checker, or general reachability verdict. The experiment therefore measures whether deterministic code-structure facts improve this audit workflow. It does not test every static-analysis technique.

Method

The target was an anonymized production proof-of-stake consensus client written in Go. Every case used the same audited build.

The benchmark started with 168 cases: 128 questions about a named potential issue and 40 open-ended review prompts. Three GPT-5.6 models, Sol, Terra, and Luna, ran at xhigh effort with the static-analysis MCP tools on and off. In the tools-off setup, the server, tool declarations, and prompt references were removed. Every setup used Luna at high effort for supporting code research.

The main comparison uses the 134 cases for which every model returned a usable verdict both with and without the MCP:

Cases	Count
Known bugs	55
Cases without a real bug	79
Cases compared	134
Cases excluded	34

All 34 excluded cases were cases without a real bug and lacked a usable tools-off verdict for at least one model. Comparing the same 134 cases gives every model and setup the same denominator. It also changes which false-positive cases are included, so the exclusion remains visible.

The metrics were:

• Recall: the share of the 55 known bugs that the model found.
• False-positive rejection: the share of the 79 cases without a real bug that the model correctly dismissed.
• Precision: the share of issues reported by the model that were real. The number of reported issues differs by setup, so precision is a descriptive comparison.
• Balanced accuracy: the mean of recall and false-positive rejection.

Each case has one run with the tools and one without. Exact McNemar tests compare which cases changed between the two setups. The 95% intervals quantify uncertainty across cases; they do not measure variation between repeated model runs.

The normalized captures are available as the full metrics CSV and the comparison-interval CSV.

Static analysis did not improve recall

Across all three models, enabling the static-analysis MCP did not improve recall.

Model	MCP tools on	MCP tools off	Off − on	Exact p	95% interval
Sol	38/55 (69.1%)	39/55 (70.9%)	+1.8 pp	1.000	[−6.1, +9.8]
Terra	32/55 (58.2%)	34/55 (61.8%)	+3.6 pp	0.727	[−6.4, +13.7]
Luna	32/55 (58.2%)	32/55 (58.2%)	0.0 pp	1.000	[−8.7, +8.7]

Sol and Terra found one and two more known bugs, respectively, with the tools off. Luna was unchanged. Every interval includes zero.

On this benchmark, deterministic code-structure facts did not help any of the three models find more known bugs. This result does not cover static-analysis systems that provide security-specific outputs such as taint paths or attacker reachability.

Precision increased by reducing false positives

With the MCP tools on, all three models correctly dismissed more of the 79 cases that had no real bug.

Model	Correctly dismissed, tools on	Tools off	On − off	Exact p	Precision, on	Precision, off
Sol	70/79 (88.6%)	69/79 (87.3%)	+1.3 pp	1.000	38/47 (80.9%)	39/49 (79.6%)
Terra	70/79 (88.6%)	66/79 (83.5%)	+5.1 pp	0.289	32/41 (78.0%)	34/47 (72.3%)
Luna	67/79 (84.8%)	63/79 (79.7%)	+5.1 pp	0.219	32/44 (72.7%)	32/48 (66.7%)

Luna showed the clearest direction: five false positives were dismissed only with the MCP, while one was dismissed only without it. The 95% interval for the tools-on difference was [−0.9, +11.0] percentage points and includes zero.

Precision increased because fewer false positives entered the tools-on results. The increase was 1.3 percentage points for Sol, 5.7 for Terra, and 6.1 for Luna. Precision uses a different number of reported issues in each setup, so these percentages describe the observed runs rather than a case-for-case statistical test.

Combining recall and false-positive rejection into one score hides what changed. Balanced accuracy with tools on versus off was 78.8% versus 79.1% for Sol, 73.4% versus 72.7% for Terra, and 71.5% versus 69.0% for Luna. The separate measures show that the gains came from dismissing false positives, not from finding more known bugs.

Frequent tool use increased cost, not recall

The models used the MCP tools very differently across the full 168-case benchmark.

Model	Audit runs using MCP tools	Calls per run	Median calls	LLM cost increase
Sol	128/168 (76.2%)	8.01	7.0	+13.2%
Terra	54/168 (32.1%)	1.25	0.0	+1.1%
Luna	162/168 (96.4%)	10.11	9.5	+14.3%

Terra did not use the MCP tools in 110 of 168 tools-on runs, so its two setups often followed nearly the same workflow. Luna used the tools in almost every run and had exactly the same recall with the tools off. A model calling a tool frequently is not evidence that the tool improved its answer.

The two models that used the MCP most also had the largest LLM cost increases: Luna +14.3% and Sol +13.2%. Terra used it least and cost rose 1.1%.

The analysis MCP ran locally and had no API fee. The cost column includes LLM usage from each model and its supporting Luna calls; local analysis compute is excluded. For Terra and Sol, the tools-off costs use a reconstructed full-168 estimate because the original tools-off coverage for cases without a bug was incomplete. The price table was verified on 2026-08-04.

Wake response-format note. This measurement is separate from the Go benchmark above. In one representative Wake MCP response, changing the same response from structured JSON to plain text reduced the o200k_base token count from 199 to 83, a 58.3% reduction. It measures response formatting in Wake, not benchmark cost or model performance.

The code-writing task also showed no improvement

A separate proof-of-concept task asked each model to write executable Go demonstrations for ten pre-registered bugs. Three models across ten bugs produced 30 runs in each setup.

Outcome	MCP tools on	MCP tools off
Test file written, compiled, and executed	26/30	27/30
Test failed on the vulnerable code	11/30	12/30
Test targeted the named package	20/30	22/30

For failing demonstrations, three cases succeeded only with the MCP and four only without it; exact McNemar p = 1.000. This task required navigating a real package and writing code that compiled, but the tools produced no measured improvement.

Across all 60 runs, 38 returned a confirmed verdict, but 16 of those confirmations had no working demonstration. A failing test is stronger evidence than a prose claim, although a human still needs to verify that the test failed for the claimed reason.

The open-ended review task was also flat. Ten runs per setup were blind-judged against 45 known bugs. The MCP-tools-on runs recovered six and the tools-off runs recovered five. Five bugs appeared only with the tools and four only without them; exact p = 1.000. The judge was too lenient to quote absolute recall reliably, so this result supports only that no difference was measured between the two setups.

What a follow-up can test

The experiment separates two questions:

1. Did the models use the static-analysis MCP tools?
2. Did the tools improve recall or precision?

The first answer was yes. Luna used the MCP in 96.4% of tools-on runs. The second answer was mixed: recall did not improve, while observed precision was higher for all three models.

A follow-up can test deterministic, security-specific outputs against audit questions they should directly help. Examples include taint paths, attacker-controlled reachability, state-transition summaries, storage-layout hazards, and candidate invariants. Each output can be tested with the tools on and off against cases where it has a clear route to the answer.

The surrounding workflow matters too. Tool declarations, response shape, context compaction, parallel-call policy, and prompts determine whether deterministic facts reach the final verdict or merely add calls and tokens.

Limitations

• The target is one anonymized Go proof-of-stake consensus client and one benchmark workflow.
• Each case has one run in each setup. Four cases were repeated separately and one verdict changed; the benchmark does not estimate run-to-run variation.
• The main comparison excludes 34 cases without a real bug. Only 33 of the original 113 cases without a real bug were externally adjudicated; the remaining labels came from production behavior.
• The MCP provides deterministic code structure, not a vulnerability verdict or a security-specific reachability result.
• Terra and Sol tools-off costs use a reconstructed full-168 estimate rather than 168 independent tools-off runs.
• The proof-of-concept scorer observes compilation, execution, and failure mechanically; a human still has to confirm why a test failed.
• The open-ended judge can compare the two setups but cannot support a reliable absolute recall claim.
• The result does not transfer automatically to other models, codebases, prompts, MCP tools, or audit tasks.

Conclusion

Static-analysis MCP tools gave non-deterministic models deterministic code facts on demand. They did not make the models’ reasoning deterministic, and they did not improve recall in this benchmark. Precision was higher for all three because the models dismissed more false positives. LLM cost rose.

The MCP is best understood as a stable input to a variable reasoning process, not as a guarantee of better audit results. Deterministic navigation and verification can still be useful, but tool use alone is not evidence of improved outcomes. Measure recall and precision separately, then test security-specific outputs against the decisions they are designed to change.