Passing attempts
| Model | 5/5 | 1–4/5 | 0/5 | |
|---|---|---|---|---|
| GPT-6 Astra | 24 | 30 | 96 | |
| Claude Opus 5 | 15 | 38 | 97 | |
| Claude Fable 5.1 | 6 | 50 | 94 | |
| Gemini 3.8 Flash | 9 | 30 | 111 | |
| GPT-5.6 Sol | 14 | 18 | 118 |
Leaderboard
25.9%
highest pass@1 estimate
GPT-6 Astra
24/150
most 5/5 cases · GPT-6 Astra
23
never solved by any model
| Model | Lab | pass@1 (%) | 5/5 cases | pass@5 (%) | Checks | Never solved | |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAI | OpenAI | 25.9 | 24 | 36.0 | 76.8 | 96 |
| 2 | Claude Opus 5Anthropic | Anthropic | 22.1 | 15 | 35.3 | 71.2 | 97 |
| 3 | Claude Fable 5.1Anthropic | Anthropic | 19.7 | 6 | 37.3 | 73.7 | 94 |
| 4 | Gemini 3.8 FlashGoogle | 15.6 | 9 | 26.0 | 59.2 | 111 | |
| 5 | GPT-5.6 SolOpenAI | OpenAI | 13.6 | 14 | 21.3 | 62.9 | 118 |
Pass Rate by Attempt Count
- GPT-6 Astra
- Claude Opus 5
- Claude Fable 5.1
- Gemini 3.8 Flash
- GPT-5.6 Sol
| Model | @1 | @2 | @3 | @4 | @5 |
|---|---|---|---|---|---|
| GPT-6 Astra | 25.9 | 30.5 | 32.9 | 34.7 | 36.0 |
| Claude Opus 5 | 22.1 | 28.4 | 31.7 | 33.9 | 35.3 |
| Claude Fable 5.1 | 19.7 | 27.9 | 32.3 | 35.2 | 37.3 |
| Gemini 3.8 Flash | 15.6 | 20.3 | 22.7 | 24.4 | 26.0 |
| GPT-5.6 Sol | 13.6 | 16.5 | 18.6 | 20.1 | 21.3 |
Source & consistency
pass@1 by source
| Model | ChestX-Det112 | NIH CXR1414 | VinDr10 | GRAZ7 | RSNA3 | FracAtlas4 |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 23.2 | 45.7 | 28.0 | 37.1 | 0.0 | 25.0 |
| Claude Opus 5 | 25.4 | 0.0 | 10.0 | 22.9 | 66.7 | 5.0 |
| Claude Fable 5.1 | 21.1 | 0.0 | 10.0 | 40.0 | 66.7 | 5.0 |
| Gemini 3.8 Flash | 11.4 | 27.1 | 20.0 | 54.3 | 0.0 | 25.0 |
| GPT-5.6 Sol | 8.6 | 32.9 | 18.0 | 48.6 | 26.7 | 5.0 |
What counts as a pass?
Findings
The model identifies which findings are present or absent and answers any questions about their location. Every checklist answer must match the reference answer for that image.
Localization
The model places a box around each required finding. Boxes are checked for overlap, position and size. Each case limits how many extra, unmatched boxes are allowed.
Diagnosis
The grader matches accepted diagnosis terms and affirmation or denial. This is lexical matching, not disease-level diagnostic validation; the free-text summary is not scored.
Next step
The selected action must belong to the case's accepted set. This deterministic membership check is not clinical adjudication: urgent_review is accepted on 148 of 150 keys, so urgency discrimination is weak.
Protocol
- Rollouts
- 5 per study
- Sampling
- temperature 0, 65,536 max tokens
- Reasoning
- xhigh (OpenAI and Claude); high (Gemini)
- Inference
- Prime Intellect Inference (api.pinference.ai)
- Images
- 1024 × 1024 px, one per study
Never solved
23/150 studies never passed: 19 ChestX-Det, 2 NIH ChestX-ray14, 1 FracAtlas, 1 VinDr-CXR.
Source terms · Images and gold are not distributed.