Mitosis Bench

1 Intro and Background

Introducing my first, of many, reinforcement learning environment and benchmark made for Prime Intellects RL residency. I wanted to create a medical environment since I haven’t seen many out there. I figured pathology would be an interesting one, we get told quite often that models will cure diseases and make vaccines. Throwing compute so that we can hillclimb medicine is one of the best, if not the best, uses of reinforcement learning. A world in which doctors, scientists, and nurses are 100x more productive due to super intelligence is a world I see very soon, and in my opinion, a world that is much better than one with just more productive engineers.

Mitosis Bench. A pathology benchmark where an agent has to find every mitotic figure in a tumor image. Claude Opus 5.5 leads all models with 28.6% pass@1.

What is a Mitotic figure? A mitotic figure is a cell caught in the act of dividing, visible under the microscope. Pathologists use mitotic figures as an indicator of how fast a tumor is growing.

(a) mitotic figures

(b) look-alikes

About the Dataset: The dataset is MIDOG++(https://www.nature.com/articles/s41597-023-02327-4) , which is an extension of the MIDOG (Mitosis Domain Generalization) challenges. I chose a mitosis dataset specifically because I anticipated that models would struggle with identifying really small figures and navigating data where the amount of look-alikes outnumber real figures. MIDOG++ has 503 tumor regions of 7 tumor types: breast carcinoma, lung carcinoma, lymphosarcoma, neuroendocrine tumor, cutaneous mast cell tumor, cutaneous melanoma, and (sub)cutaneous soft tissue sarcoma. This dataset combines both human and dog histological specimens and is labeled by expert consensus.

2 Task and Reward shape

When designing the input and output I gave the agent a 768 x 768 px tissue image, sometimes I gave it two side by side. The model would NEVER get told how many cells there are, it is evaluating it similarly to how a pathologist would in a real work environment. The agent would output its answer in the format of “ Answer: [[x,y], …] on a 0-1000 grid.

(a) input

(b) expert labels

Reward: I used a binary reward, it would only get the reward if the answer matches exactly the right cells within 30 px (7.5 µm which is the MIDOG challenge’s radius) with no extra or missing cells. No partial credit since we don’t have a judge and I could see partial credit opening up some reward hacks for the models. I still recorded precision and recall accuracy so we could diagnose failure. The model gets no tools; all it can do is inspect the image (crop, zoom, enhance) with whatever its harness offers. Rather than using the entire dataset I selected only images where the experts agreed on all the labels and where there were more than three figures, this was my eval set. The benchmark has 100 tasks across 55 different microscope slides and 7 different tumor types, the images in the benchmark were randomly seeded from the eval set.

3 Running the Evals

All the models ran through a Pi harness on high reasoning effort. For each task the models had 5 attempts and I only allowed retries if there was an infrastructure failure. I ended up having to retry most of deepseeks attempts since I ran into a bit of an interesting issue during evaluation, 147 attempts failed with a “Downloaded image content cannot exceed 30MB failure”. After running some tests I found out that the reason behind this was that agents saved zoomed PNG crops and since the harness would resend every image on every turn the requests piled up. To fix this I added a small local relay that re-encoded non-JPEG images as JPEG. The encoding was deterministic so caching still works and the largest request dropped to 17.5 MB with zero rejections across all its runs. To prove that this didn’t affect the eval results I measured performance with/without the relay, I used Opus as the control group (since it never hit the limit) and its score moved only +0.9 points. Deepseek ended up rising from 6% to 8.8% because the cap failures disappeared. In order to make sure there was no cheating I scanned every single tool output for all attempts for answers, scoring text or ids and found nothing.

4 Results

Something interesting to note from the chart is that Opus stops very early, especially on tasks it fails. In 334 out of 500 of its attempts it lists too few points. Deepseek makes 68 model calls per attempt while Opus only makes 6! Another interesting thing is that of the 100 tasks, Opus never solves 62 of them but there are 19 tasks it solves every time. On the other hand, 38% of tasks GLM attempted had some attempts pass and others fail. By pass@5 GLM is basically caught up with Opus! The mixed result tasks are perfect for RL.