As frontier models become increasingly more capable, the need to ensure security upon deployment grows. Pattern Labs is proud to share that we played a significant role in assessing OpenAI's o3 and o4-mini's cybersecurity capabilities through a comprehensive evaluation, as referenced in the models’ System Card.
Leveraging some of our evaluation suite and frameworks, we assessed the cybersecurity offensive capabilities of both models.
We conducted quantitative and in-depth qualitative analyses with multiple key conclusions. Both models showed some strategic planning. For example, when given source code for a target system, the models would occasionally compile the code themselves to create a simulated version of the target that’s easier to test or explore. Despite that, in other cases, the models failed to reliably recognize the progress they had made in previous steps. In some instances, for example, after finding a key piece of information essential in obtaining further access, they would completely ignore it.
These and other limitations contributed to the models’ inability to solve hard challenges (see Solve Score blog post) and led us to conclude that despite the performance improvements, o3 and o4-mini do not pose a significant risk as a cyber-offensive tool.
As the AI landscape evolves, Pattern Labs remains committed to pioneering security evaluation methodologies that stay ahead of potential threats. Our collaboration with OpenAI on the o3 and o4-mini evaluation represents just one facet of our broader mission to establish industry standards for secure AI deployment. By combining technical rigor with real-world simulated environments, we're helping to build a future where powerful AI systems can be safely deployed.
@misc{pl-openais2025,
title={Pattern Labs' Role in OpenAI's o3 and o4-mini's Security Evaluation},
author={Pattern Labs},
year={2025},
howpublished={\url{https://patternlabs.co/blog/openais-o3-and-o4-mini-security-evaluation}},
}