Physician-graded clinical data for frontier AI.
Practicing physicians write net-new clinical cases and grade how a model works through them. You get faithful signal for medical reasoning, not desk labeling, simulated patients, or a model grading itself.
The data
A catalog built to fix what today's benchmarks can't
Six physician-produced data tracks, each positioned against the gold-standard benchmark it improves on.
Interactive clinical cases + trajectory grading
Physician-authored sequential patient cases with a hidden ground-truth state and faithful transitions, plus physician grading of a model's whole decision trajectory.
Physician rubric-graded conversations
Net-new consented multi-turn clinical conversations graded by physicians against adjudicated rubrics for correctness, completeness, safety, and calibration.
Clinical reasoning + physician reasoning traces
Vignettes, differentials, and management plans with step-wise physician reasoning traces. This is supervised signal for medical verifiers and process-reward models.
Multimodal diagnostic imaging
Physician diagnostic grounding and reasoning across radiology, dermatology, pathology, ECG, and clinical photographs.
Egocentric procedural / surgical video
Consented first-person procedural video with dense hierarchical expert annotation (procedure → phase → step → action), operator rationale, and an error taxonomy.
Safety, red-teaming & hallucination
Physician-authored adversarial cases with a severity-graded harm/error taxonomy that separates reasoning-process failures from factual and safety ones.
The problem
Most clinical evals are static, contaminated, and model-judged
Each of those is a measured failure mode, not an opinion.
→ 51.6%
GPT-4 falls from ~90% on the MedQA exam to 51.6% once the same medicine is played out as a sequential encounter; Llama-2-70B drops from ~60% to 4.5%.
AgentClinic, npj Digital Medicine, 2026
AUC 0.49-0.66
LLM judges separate complete from incomplete clinical answers only marginally above chance. Even when they agree with a clinician, they cite the same reasoning just 24.6% of the time.
Independent evaluations of LLM clinical graders, 2025 to 2026
< 32%
No frontier model scored above 32% on HealthBench Hard at release, a benchmark that took 262 physicians writing 48,562 rubric criteria to build.
HealthBench, OpenAI, 2025
How it works
Produce-and-grade, not labeling
The state and the grades both come from clinicians, so the signal is faithful and hard to game.
Physicians author
Practicing physicians build each case as an interactive text (or multimodal) environment with a hidden ground-truth state and faithful transitions for every clinically sensible action.
Models act
The model takes a history, orders and reads tests, revises the differential, and manages the patient over time. Each fact appears only in response to an action.
Physicians grade the trajectory
The same physicians grade the whole decision process against an adjudicated rubric: accuracy, calibration under uncertainty, and safety at each step.
Delivered in your formats
You get it as an evaluation, and the graded trajectories double as training signal for SFT, preference/RLHF, and process-reward or verifier training.
Why physicians, not simulation
Fidelity is the whole game
Automated patient simulators and model judges don't reach the bar the benchmarks' own authors set. Physicians who author the case and grade the trajectory do. A general labeling workforce doesn't.
Consent & ethics
Physicians write net-new cases. Nothing comes from identifiable records, so the content carries no patient-level privacy burden.
Uncontaminated
Benchmarks built from published case reports can already sit in a model's training data. Net-new cases cannot.
Compliance-ready
If you later fold in real cases, the usual consent, ethics approval, de-identification, and HIPAA/GDPR-aware licensing apply.
See a case before you commit
Book a demo and we'll build one case end to end: the environment, the hidden ground-truth state, the grading rubric, and a graded model rollout, in your evaluation format, at no cost. Your team judges the fidelity first.