In the rapidly evolving world of frontier models, rigorous security assessment remains paramount. We're proud to share that Pattern Labs played a significant role in the comprehensive security evaluation of Anthropic's recently released Claude 3.7 Sonnet model.
As referenced in Claude 3.7 Sonnet's system card, our state-of-the-art cyber evaluation suite was used to assess the model's cyber capabilities through our design, implementation and analysis. We served as the "expert partners" mentioned throughout Anthropic's documentation, evaluating potential dangerous cyber capabilities and finding it responsibly secure.
"To test the model cyber capabilities, we developed a series of realistic cyber challenges in collaboration with expert partners [12]. We designed these challenges to cover a range of cyberoffensive tasks."
The evaluation methodology utilized the SOLVE scoring system, which we pioneered—an open standard developed by Pattern Labs researchers and adopted by Anthropic, and others, for comprehensive assessment. The SOLVE system assigns challenges a difficulty score from 0.0 to 10.0 based on multiple factors, including:
These components are combined to form an overall difficulty score, and then challenges are categorized as Easy (0.0-3.9), Medium (4.0-6.9), Hard (7.0-8.9), or Expert (9.0-10.0). For an in-depth description, see our blog post.
Pattern Labs is developing multiple evaluation types that closely simulate real-world scenarios, going beyond theoretical exercises to test practical capabilities:
"We focus on network and cyber-range challenges as key indicators for catastrophic risk, testing comprehensive attack capabilities from reconnaissance to exfiltration."
The network attack simulation challenges in particular represent some of our most sophisticated evaluations, requiring comprehensive capabilities across the entire cyber kill chain.
Advancing Frontier Model Security Research
Our work with Anthropic exemplifies the importance of the intersection of cybersecurity expertise and frontier AI security. As these models become increasingly capable and more widely used, developing robust, standardized evaluation methodologies remains essential for the research community. Pattern Labs continues to refine our evaluation frameworks through rigorous research, contributing to the scientific understanding of AI security, as well as working alongside frontier labs and governments to create the mitigations required to be prepared for the time that evaluations show elevated risk levels. We believe transparent collaboration between security researchers and AI developers helps ensuring that tomorrow's most powerful models are developed responsibly with appropriate safeguards.
@misc{pl-claude2025,
title={Pattern Labs' Role in Claude 3.7 Sonnet Security Evaluation},
author={Pattern Labs},
year={2025},
howpublished={\url{https://patternlabs.co/blog/claude-3.7-sonnet-evaluation}},
}