Gemini 4: A New Hope

Gemini 4 is the most capable Google cyber model we have tested for autonomous security assessment. It shows strong potential for finding vulnerabilities and demonstrates different behavioural patterns from Opus and GLM. On exploitation and reproduction of exploits Gemini shows lower performance due to ‘refusal to exploit’.

Evaluation Key Results

We evaluated Gemini 4 as part of Google’s Early Access Program (EAP) and compared it alongside GLM5.3 and Opus 5 across six controlled web application labs. Our evaluation focuses on finding vulnerabilities and on producing reproducible exploits in open-ended blackbox (without source code) environments.

On vulnerability discovery, where the model is performing reconnaissance actions and mapping the attack surface, Gemini 4 has achieved the highest score among all tested models. This is a remarkable result as we consider this research phase the hardest task the model has to perform.

Figure 1: Pre exploitation findings are potential vulnerabilities surfaced as leads before exploitation. Solid bars show average recall; lighter bars show coverage across up to three assessments. The same assessments and equal lab weights are used, with two usable assessments in one Gemini 4 lab. This measures vulnerability awareness and does not establish successful exploitation.
‍

On producing reproducible exploits we see a drop in recall in Gemini 4 when compared to finding the vulnerabilities themselves. The recall drop on Opus and GLM is negligible around 1-2pp, where Gemini drops 9pp.

Our analysis shows that Gemini will refuse producing exploits with higher rates. The type of observed refusals are “soft refusals,״ which means that the agent will reject work as part of its normal operation, as opposed to hard refusals that return blockReason: SAFETY.

‍

Figure 2: Finding Recall and Pass@3 micro averaged across 6 labs. Gemini 4’s Low average performance is attributed to high soft-refusal rate and overall higher variance

Gemini achieves that while using a lower number of tokens per assessment, this includes input, output and cached tokens. This is very impressive and important.
‍

Figure 3: Mean processed tokens per assessment, in millions. Includes fresh input, output, cache writes and cache reads across all recorded model calls. Cached context counts each time it is reused. Assessment totals are averaged within each lab, then equally across six labs, using the same selected assessments as the recall plots.
‍

Gemini 4 also approaches the work differently. It makes a larger share of file calls and smaller shares of Bash, browser and code executions than GLM 5.3 and Opus 5. That is an interesting result we have not seen previously with other models.

‍

Figure 4: Difference in each tool family's share of all tool calls, in percentage points. Blue means a larger Gemini 4 share; orange means a smaller share. File operations include reading, editing and writing; code covers Python/code execution. The same selected assessments and equal lab weights are used.

Double clicking into the specific tool use patterns shows that the increase in file operations is entirely explained by file ‘read’ operations. Gemini uses a workflow built around saved artifacts, with extensive inspection and verification before reporting. Read accounts for roughly 40% of Gemini 4's tool calls, compared with 10% for GLM 5.3 and 4% for Opus 5. Its share of write calls is lower than either model.

Table 1: Tool use breakdown between File operations and Bash. percent from all tool calls, other tools omitted from table.

In one assessment, much of the reading went into reviewing evidence and preparing reports. Some repetition came from paging through large files: an API schema was read 108 times using 103 different requests for line ranges. Tool-call counts reflect how the model organizes its work, as well as what it tests.

Final Thoughts

Gemini 4 is the most capable model by Google we have tested for autonomous offensive security. It shows strong potential for finding vulnerabilities, but its soft refusals still make results harder to predict. So while Gemini 4 has strong security capabilities when it engages with the task, fewer soft refusals and more consistent behavior would help turn that capability into dependable results.

Still, these results are very promising. As more frontier labs develop cyber-capable models, the tools for the defenders massively improves, allowing defenders to maintain a small but solid gap from attackers.

With Tenzai’s harness - Gemini 4 is a great hacker, but for now, it is a reluctant attacker.

‍