Performance of a large language model on the reasoning tasks of a physician
Multiple-choice medical exams no longer separate strong AI systems from weak ones, so a team led by Harvard Medical School and Beth Israel Deaconess Medical Center asked a harder question: can a reasoning model handle the open-ended judgements physicians make every day? They ran five experiments with OpenAI's o1 model, covering differential diagnosis, explanation of diagnostic reasoning, triage, probabilistic reasoning and management planning, and had practising physicians grade the outputs with validated scoring tools. The test set included 143 clinicopathological cases from the New England Journal of Medicine, difficult simulated cases, and a real-world comparison on 76 randomly selected patients from a Boston emergency department, where the model and two attending physicians each read the same chart at three points: triage, initial evaluation and the admission decision. The model matched or exceeded physicians across most tasks. At triage, when only vital signs and a brief nursing note were available, it gave an exact or very close diagnosis considerably more often than the physicians; once more information arrived, the gap narrowed and was no longer statistically significant.
Key findings
- On 143 NEJM case conferences, the reasoning model gave the correct or very close diagnosis clearly more often than GPT-4 did.
- In the emergency-department test, the model's advantage was largest at triage, when information was scarcest; blinded reviewers could not reliably tell AI output from physician output.
- The authors stress that the model only read text written by others and call for prospective clinical trials before any deployment.
Why it matters
Hospitals in the Gulf are investing heavily in digital health, and this is the first peer-reviewed evidence in a top journal that a general-purpose reasoning model can rival specialists on real emergency cases. It argues for AI as a second opinion at the moments when clinicians have the least information, while making clear that trials, governance and physician oversight must come first.
