Which decision model codes charts best?

Six decision models answered the same – graded questions about – real charts from – RapidClaims clients. The questions are the decisions a coder makes: visit type, billable or not, E/M level and its MDM parts, ICD-10 and CPT codes, modifiers, time and more. Every answer is checked against a graded key.

Leaderboard

Score is the share of questions answered correctly, counted only on questions every model answered, so all models face the same set. The dashed line is a second grader who answered blind, a rough ceiling for how often two careful readers agree.

Yes/No and Choice: accuracy by question type. Chart all right: charts where the model got every question right. Coverage: questions the model answered at all; long charts some models refuse. Brier: error of its yes/no probabilities; lower is better. Latency: median seconds per request of ~14 questions, on the hardware listed in Method.

By coding area

Accuracy on each family of questions. Darker green is better; the last column is the second grader. Sorted by how many questions the area has.

By client

The same questions split by client. Clients differ in specialty, EHR and how their charts read, so this shows where a model holds up and where it doesn't.

Where it gets hard

Each panel splits the questions one way. Graders tagged every question easy or hard, and the answer as clear in the note or a judgement call. Production prompts are questions RapidClaims' workflows send to Jev (live or in development); new questions were written for this benchmark. Disputed means the graded answer differs from the code the client's coder billed.

Can you trust its confidence?

Every yes/no answer comes with a probability. A well-calibrated model that says 90% is right about 90% of the time, so its line hugs the diagonal. This matters for automation: confidence is what decides whether a chart goes straight to billing or to a coder.

Head to head

Each cell counts the questions the row model got right and the column model got wrong. Hover a cell to see how often the two give the same verdict.

Question explorer

Questions written for this benchmark and asked in the same words on at least 5 charts, with each model's accuracy on them. Sort by spread to find where models disagree most. Only the generic wording is shown, never chart text; production prompts are scored but not shown.

How this was measured

The questions and the key

858 charts were drawn from 17 RapidClaims clients: emergency and hospitalist, primary care, OB/GYN, urology, GI, dental, PACE, anesthesia and ambulance. Each chart was graded in three independent passes. A wrote the questions a coder must decide for that chart and answered them with evidence. B answered the same questions blind. C settled every item where A and B differed, B flagged the question, or both differed from the coder's billed code. A and B agreed on 95.1% of questions. 23 questions C found unanswerable were dropped.

The graders are AI graders working from the chart text, the client's coding rules and public coding guidelines. They are not certified coders. Where the key differs from what the client's coder billed (7.5% of questions, almost all ICD, CPT and E/M level), it is marked disputed and kept. The "agreed vs disputed" panel shows each model on both.

Scoring

A yes/no answer counts as yes when the model's probability is at least 0.5. A choice answer is the model's top option. Every model got the whole chart text in the same request format (the System One API: a state and up to 16 questions per request) and no examples, prompt tuning or client rules. A question a model refused, usually because the chart was longer than its window, counts toward its coverage, not its score. The headline score uses only questions every model answered.

Things to keep in mind

9,084 of the questions come from RapidClaims' workflows (live or in development), which were written and tuned against Jev's answers. That favours Jev on those questions; the "production prompts vs new questions" panel shows by how much.
Open models ran at their published defaults, except that windows were raised to 64k tokens where the default would have cut a long chart. No model was fine-tuned on these charts.
Latency depends on the hardware, the batch size and how busy the server was. Read it as a rough ordering, not a spec.
The charts contain patient information, so this site shows only aggregate scores and the generic wording of questions written for this benchmark. Clients appear as letters with a broad specialty.