Changelog
- 23 September 2026: Initial version.
1 Introduction
Basin is a security research model for authorized vulnerability discovery. It can inspect a live target or a supplied codebase, investigate a suspected weakness, and produce a proof of concept and a written finding for human review.
| Attribute | Basin |
|---|---|
| Model | Dense language model with fewer than 50 billion parameters |
| Context window | 256k tokens (262,144 tokens) |
| Input and output | Text input; text and tool calls as output |
| Primary use | Authorized vulnerability research and verification |
| Evaluation mode | Information available to the agent |
|---|---|
| Black-box | An in-scope URL, host, or service endpoint |
| Source-assisted | An in-scope repository or package tree, with a test target when the task requires one |
Basin was developed through pretraining and post-training. The model is intended to assist authorized researchers, not to make unattended decisions about production systems. Operators must define scope, review findings, and control access to targets and tools.
2 CyberGym
CyberGym Level 1 tests whether an agent can reproduce a known vulnerability from a description and an unpatched codebase. Its official verifier checks that the submitted proof of concept triggers on the vulnerable version and not on the fixed version. The suite contains 1,507 cases across 188 projects. CyberGym benchmark
Basin was evaluated in one pass over the 1,507 cases using a Codex CLI agent harness. The case-level score summary records 1,218 successful cases (80.82%); 21 cases were not scored.
Peer scores are published by the CyberGym leaderboard (Grok 4.6), Anthropic (Claude Opus 4.8), and Meta (Muse Spark 1.1). They use different agent harnesses and evaluation settings. The chart provides context, rather than a matched head-to-head ranking.
3 General capability evaluations
These public suites check broader coding and instruction-following behavior. The runs used the target BF16 model on a 32,768-token serving window; all evaluated prompts fit within that window.
| Evaluation | Basin | Protocol |
|---|---|---|
| HumanEval pass@1 | 97.6% (160 / 164) | Full suite; temperature 0; xhigh thinking; Python execution in Docker |
| IFEval final accuracy | 82.3% | Full 541 prompts; xhigh thinking |
| IFEval prompt strict / loose | 77.3% / 80.8% | Automatically checked constraints |
| IFEval instruction strict / loose | 84.2% / 86.8% | Automatically checked constraints |
4 Dual-use knowledge evaluations
WMDP scores measure answers to multiple-choice questions about potentially sensitive knowledge. They do not measure vulnerability discovery, operational ability, or refusal behavior. These runs used direct-answer mode at temperature 0, with no invalid responses or output-cap hits. WMDP benchmark
| Evaluation | Basin |
|---|---|
| WMDP-Cyber accuracy | 65.5% (1,302 / 1,987) |
| WMDP-Bio accuracy | 82.9% (1,055 / 1,273) |
| WMDP-Chem accuracy | 66.4% (271 / 408) |
5 Refusal robustness
StrongREJECT scores responses to forbidden requests; lower scores indicate less disallowed assistance. The full set has 313 prompts. The original baseline and the replication use different judges, so they are reported separately. StrongREJECT benchmark
| Evaluation | Basin | Protocol |
|---|---|---|
| Unmodified baseline | 0.029 | Full 313 prompts; xhigh thinking; independent GPT-4o judge |
| Unmodified baseline jailbreak rate | 0.6% (2 / 313) | Same GPT-4o-judged run |
| Unmodified baseline replication | 0.0192 | Full 313 prompts; xhigh thinking; GPT-4.1-mini judge; run manifest |
| Wikipedia prompt transformation | 0.0032 | Full 313 prompts; same GPT-4.1-mini judge; run manifest |
The transformation score is the mean across all 313 cases, with no best-of aggregation. These results do not establish robustness to every jailbreak or to long conversations.
6 Intended use and limitations
Basin is intended for authorized penetration testing, source-assisted review, reproducible proof of impact, and defensive verification. It is not intended for attacks outside an agreed scope, credential theft, malware deployment, or other criminal use. Human reviewers should verify the proof and decide whether a finding is actionable.
The comparison chart combines provider-reported results from different harnesses. Benchmark performance does not establish safety on live networks or guarantee that a generated finding is correct. Dual-use skills require access controls, authorization, and logging outside the model.