Real-SWE 1.1
Benchmarking frontier AI models on private, real-world, enterprise codebases.
Read the blogLeaderboard
October 2026- Rank 1
Claude Opus 5.5Resolution rate: 56.80%, one standard error either side: 53.5% to 60.1% - Rank 2
GPT 6 AstraResolution rate: 56.00%, one standard error either side: 54.4% to 57.6% - Rank 3Claude Sonnet 5.5Resolution rate: 51.20%, one standard error either side: 49.0% to 53.4%
- Rank 4GPT 6.1 SolResolution rate: 47.20%, one standard error either side: 44.8% to 49.6%
- Rank 5Claude Fable 5.1Resolution rate: 45.60%, one standard error either side: 42.5% to 48.7%
- Rank 6
Muse Wonder 1.0Resolution rate: 42.40%, one standard error either side: 39.9% to 44.9% - Tied for rank 7
Gemini 4 ArgonResolution rate: 39.20%, one standard error either side: 35.9% to 42.5% - Tied for rank 7
Kimi K3.1 ProResolution rate: 39.20%, one standard error either side: 35.8% to 42.6% - Tied for rank 9
Particle-1Resolution rate: 38.40%, one standard error either side: 35.9% to 40.9% - Tied for rank 9
DeepSeek V4.1 FlashResolution rate: 38.40%, one standard error either side: 35.8% to 41.0% - Rank 11
Hy4 previewResolution rate: 33.60%, one standard error either side: 30.9% to 36.3% - Rank 12
Grok 4.7Resolution rate: 28.00%, one standard error either side: 25.3% to 30.7% - Rank 13
GLM-5.3Resolution rate: 26.40%, one standard error either side: 23.9% to 28.9% - Rank 14
Qwen3.8-Max primeResolution rate: 24.00%, one standard error either side: 21.2% to 26.8% - Rank 15GPT 6 LunaResolution rate: 10.40%, one standard error either side: 8.5% to 12.3%
- Rank 16
InklingResolution rate: 8.00%, one standard error either side: 6.5% to 9.5% - Rank 17
Seed 2.1 TurboResolution rate: 5.60%, one standard error either side: 4.1% to 7.1% - Rank 18
Nemotron 3 UltraResolution rate: 4.80%, one standard error either side: 3.4% to 6.2%
| Rank | Model | Resolution rate |
|---|---|---|
| 1 | Claude Opus 5.5 | 56.80%, one standard error either side: 53.5% to 60.1% |
| 2 | GPT 6 Astra | 56.00%, one standard error either side: 54.4% to 57.6% |
| 3 | Claude Sonnet 5.5 | 51.20%, one standard error either side: 49.0% to 53.4% |
| 4 | GPT 6.1 Sol | 47.20%, one standard error either side: 44.8% to 49.6% |
| 5 | Claude Fable 5.1 | 45.60%, one standard error either side: 42.5% to 48.7% |
| 6 | Muse Wonder 1.0 | 42.40%, one standard error either side: 39.9% to 44.9% |
| Tied for 7 | Gemini 4 Argon | 39.20%, one standard error either side: 35.9% to 42.5% |
| Tied for 7 | Kimi K3.1 Pro | 39.20%, one standard error either side: 35.8% to 42.6% |
| Tied for 9 | Particle-1 | 38.40%, one standard error either side: 35.9% to 40.9% |
| Tied for 9 | DeepSeek V4.1 Flash | 38.40%, one standard error either side: 35.8% to 41.0% |
| 11 | Hy4 preview | 33.60%, one standard error either side: 30.9% to 36.3% |
| 12 | Grok 4.7 | 28.00%, one standard error either side: 25.3% to 30.7% |
| 13 | GLM-5.3 | 26.40%, one standard error either side: 23.9% to 28.9% |
| 14 | Qwen3.8-Max prime | 24.00%, one standard error either side: 21.2% to 26.8% |
| 15 | GPT 6 Luna | 10.40%, one standard error either side: 8.5% to 12.3% |
| 16 | Inkling | 8.00%, one standard error either side: 6.5% to 9.5% |
| 17 | Seed 2.1 Turbo | 5.60%, one standard error either side: 4.1% to 7.1% |
| 18 | Nemotron 3 Ultra | 4.80%, one standard error either side: 3.4% to 6.2% |
We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.
Resolution rate by estimated cost per rollout
| Model | Resolution rate | Estimated cost per rollout | |
|---|---|---|---|
| 1 | Opus 5.5 | 56.80% | $13.95 |
| 2 | GPT 6 Astra | 56.00% | $3.78 |
| 3 | Sonnet 5.5 | 51.20% | $10.09 |
| 4 | GPT 6.1 Sol | 47.20% | $0.79 |
| 5 | Fable 5.1 | 45.60% | $14.24 |
| 6 | Muse Wonder 1.0 | 42.40% | $3.76 |
| 7 | Gemini 4 Argon | 39.20% | $4.66 |
| 8 | Kimi K3.1 Pro | 39.20% | $2.03 |
| 9 | Particle-1 | 38.40% | $0.56 |
| 10 | DeepSeek V4.1 Flash | 38.40% | $0.19 |
| 11 | Hy4 preview | 33.60% | $1.13 |
| 12 | Grok 4.7 | 28.00% | $2.66 |
| 13 | GLM-5.3 | 26.40% | $3.35 |
| 14 | Qwen3.8-Max prime | 24.00% | $7.06 |
| 15 | GPT 6 Luna | 10.40% | $0.08 |
| 16 | Inkling | 8.00% | $5.13 |
| 17 | Seed 2.1 Turbo | 5.60% | $3.03 |
| 18 | Nemotron 3 Ultra | 4.80% | $1.39 |
About the benchmark
Private codebases
Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
Work with business consequences
Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
Company-specific complexity
Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.
Task examples
View analysisTypeScript
Tax jurisdiction
Restore tax determination on invoices from each business's tax regime: own rate, authority quote or none, with exemptions and filing.
Solved by 4 of 18 models
Ruby
FX quote compatibility
Restore FX quotes and finish the settlement lifecycle of a Rails API.
Solved by 10 of 18 models
Go
Live-chat block history
Persistent live-chat history in sequential batches of 100 messages.
Solved by 14 of 18 models
Python
NSE and BSE instrument sync
Sync financial instruments with a trading-system feed and the NSE and BSE exchange feeds.
Solved by 1 of 18 models
One dot per model, in leaderboard order. A filled dot means the model passed the task at least once.
Evaluation setup
Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.
Read the full analysis