Periapical Lesions are pathological changes or breakdowns of tissue around the root tip of the tooth. This type of infection typically occurs at the tip (apex) of a tooth's root where the tooth's nerve and blood vessels exit into the surrounding jawbone. Dentists find them on X-rays and grade them according to the Periapical Index (PAI). These X-rays are of the whole jaw and are usually 2,400 pixels wide, but a typical lesion is usually only 60 pixels small. And agents struggle with it, the best model Gemini Carbon only achieves a 16% pass@1 and most models struggle to get past a 10% pass rate!
Task Shape
The model is put into a similar setting as a dentist. It gets a panoramic film of the jaw, a short referral note, a codebook explaining the grades, and a grading chart of example lesions for grade 3, 4, and 5 on the PAI. To submit its answer it must file a report and localize every lesion along with the corresponding PAI grade. It is sandboxed with no internet, all it can do is crop/zoom/enhance the film with python as much as it pleases. Scoring only factors two things:
- Did it box the correct place, a box counts if it overlaps the dentist’s box by atleast 30%. Extra boxes are penalized
- Did it also flag the correct PAI grade for the lesion.
Scoring is binary so in order to pass it must have found every lesion, correctly graded, with no extra boxes. There is at least one lesion in each film so there is a sentence in the prompt to let it know.

Northfield Community Dental Service Received 2026-09-08 Patient P-A6E2 Study: dental pantomograph (one exposure) Referring clinician: Dr Marchetti Reason: radiographic review
# Findings codebook — periapical reporting
Report every periapical lesion. Box the lesion itself (the radiolucency at the root tip), not the whole tooth, and grade it on the Periapical Index:
| Grade | Meaning |
| --- | --- |
| 3 | changes in bone structure with some mineral loss |
| 4 | periodontitis with well-defined radiolucent areas |
| 5 | severe periodontitis with exacerbating features |
Grades 1 (normal periapical structures) and 2 (small changes in bone structure) are not lesions and are not reported. `pai-reference.jpg` shows example lesions of each reported grade from other studies, each outlined the way a lesion is boxed.
Every study on this worklist has at least one periapical lesion.
## Report format
One JSON object: `{"lesions": [{"box": [x1, y1, x2, y2], "pai": 4}]}`.
- Box corners are fractions of the image width and height, measured from the top-left corner; keep every value between 0 and 1.
- One entry per lesion.
The Data
I used Open Source data from Hanoi Medical University (Do et al., 2024, CC BY 4.0), 3,863 panoramic films all with dentist authored gold (localized lesions and corresponding grades). This Data was split into 3,087/389/387 train/validation/test splits with near duplicate films removed and look-alikes kept in the same split.
Setup
I benchmarked six models: Gemini 4 Carbon, GPT-6.1 Sol, Claude Opus 5.5, Kimi K3.1 Pro, GLM 5.3 Flash, DeepSeek V4.1 Flash. Every model runs in the Pi harness at high reasoning effort with a 100 turn cap. I chose a random 100 films from my test split and ran 3 attempts for each model. Attempts that failed due to network, infrastructure, or provider errors were rerun.
Results
| Model | pass@1 | pass@2 | pass@3 | Mean reward | Cost per attempt |
|---|---|---|---|---|---|
| Gemini 4 Carbon | 15.7% | 21.3% | 25.0% | 0.399 | $1.77 |
| GPT-6.1 Sol | 6.3% | 11.3% | 15.0% | 0.253 | $0.15 |
| Claude Opus 5.5 | 8.3% | 10.3% | 12.0% | 0.248 | $1.58 |
| Kimi K3.1 Pro | 2.3% | 3.3% | 4.0% | 0.109 | $0.37 |
| GLM 5.3 Flash | 1.0% | 2.0% | 3.0% | 0.055 | $0.09 |
| DeepSeek V4.1 Flash | 0.3% | 0.7% | 1.0% | 0.035 | $0.05 |
Carbon clearly dominates the benchmark, although it is the most expensive, barely surpassing Opus's cost, congrats to the deepmind team on a strong model. Sol and Opus are roughly tied with Sol edging ahead towards the end at 1/10th of the cost! None of the models are close to saturating this benchmark, Carbon struggles to a 25% pass@3! Kimi K3.1 is by far the biggest loser of this benchmark spending almost three times as much as 6.1 Sol but still not able to crack a 5% pass@3 !
Failure Modes
The most common failure across all models is the wrong tooth; on the low end 114/300 of Gemini Carbon’s localizations matched no real lesions and more embarrassingly 280/300(!!) of Deepseeks reports matched no real lesions.
Kimi considers the tooth that has the lesion, then talks itself out of it:
The lesion was at tooth 36. Kimi boxed a different tooth, with zero overlap.
Often the model loses track of where its own crops are:
Another funny failure mode is with GLM 5.3 would draw very big boxes, compared to the dentists small and precise localizations. GLMs boxes had a median area 2.7 times larger than the dentists. On one attempt it found the right molar and drew a 210x200 px box around a 83x77 px lesion, getting itself a zero. Carbon Sol and Opus on the other hand was more precise, a box covered a lesion's centre but missed on size in 39 GLM attempts, 38 Kimi, 18 Opus, 14 Sol and 12 Carbon.
The models grading was skewed heavily towards the middle ,
| Model | Share of submitted lesions graded 4 |
|---|---|
| DeepSeek V4.1 Flash | 77% |
| Claude Opus 5.5 | 70% |
| GPT-6.1 Sol | 64% |
| GLM 5.3 Flash | 60% |
| Gemini 4 Carbon | 57% |
| Kimi K3.1 Pro | 47% |
Dentists on the other hand only graded 29% of lesions as grade 4! Finding a lesion and then losing on the grade was common, every matched lesion was graded wrong in 61 of Sol's 151, 57 of Carbon's 186 and 51 of Opus's 126.
It measures the lesion as too small for a 4, then calls it a 4 anyway:
(The dentist also said 4 here, so it got lucky.)
Alot of the models were also conservative with how many lesions they thought the film had. They were wrong. 42 of the 100 films had 2 or more lesions, some even have seven! Opus reported exactly one lesion in 277 of its reports!
The film has five lesions and Opus filed one:
The film has two lesions:
The film has seven lesions and Kimi filed two:
Deepseek used all 100 turns on 178 of the 300 attempts, most other models didn't even get near the turn limit! It kept telling itself it needed to make a decision… and then opened another crop, what an indecisive model!
Its last thought before the limit:
The fun part… cheating !
No models found an answer, since answers never enter the sandbox, but some of the models spent time looking anyway!
Out of ideas on the image, it turns to the harness:
followed by the command find /app/.vf-pi-agent-… -maxdepth 3 -type f | head -50; env | sort | grep -i -E 'task|study|image|report|pantom|pai'. It found the harness's own files and nothing else.
Kimi checks the image file for a hidden answer
It did this in 72 attempts. The file contains only the scanner's export header.
Guessing what the grader wants
Carbon, Kimi, GLM and DeepSeek talk about graders in most attempts. Their guesses are usually wrong.
(It is 0.3.)
Works out the expected score of reporting one lesion against two:
Sol rarely talks about graders, but here it guesses how many lesions one would expect:
It filed three. The film has one, and none of its three matched.
Decides the X-ray is fake:
The films are real patients.
Invents a benchmark:
(vf-pi is the harness's folder name.)
Some funny quirks
Kimi grades by ruler. On 29651_a2 it noticed its box was almost the same size as an example on the grading chart:
The dentist's grade was 5, and it got full credit.
Carbon plays detective. On 09351_a1 it read the corner watermark and announced:
That is the scanner's export stamp and no help with the lesions.
On 02261_a2 it calibrated the film to "roughly 0.097 millimeters per pixel" to measure lesions in millimetres.
GLM installs what it already has. On 02191_a3 it ran pip install pillow in a sandbox with no network. Pillow was already installed.
cd /tmp && python3 -c "
from PIL import Image
im = Image.open('/root/pantomograph.jpg')
print(im.size)
" 2>/dev/null || (pip install pillow -q && python3 -c "
from PIL import Image
im = Image.open('pantomograph.jpg')
print(im.size)
")Closing Notes
Once again, thank you so much to Prime Intellect and especially Sebastian for the help and guidance with building this out. It was fun to play around with the models and read their traces, I was surprised by how poorly they perform and I hope that soon we can saturate this benchmark, along with all medical/science related benchmarks 😃 Stay tuned for more, if you are interested in chatting about RL / evals or even have any book/film recs for me, feel free to email me: dtirandaz@gmail.com