ModelsArchitectures & capability
Microsoft's MAI-DxO scores 83.2% on hard NEJM cases in Science paper, versus 36.1% for generalist physicians
A peer-reviewed Science paper reports that Microsoft's MAI-DxO diagnosed 83.2% of 304 difficult NEJM cases, versus 36.1% for generalist physicians. Most of the gain comes from OpenAI's GPT-5-class models, and the test was a text-only simulation.
From the TechCrunch AI article of October 22, 2026 and cited sources: In a text-only, step-by-step simulation built from 304 NEJM case records, MAI-DxO reached 83.2% diagnostic accuracy. [7] [3] [5]
SOURCES[7] Unite.AI – Microsoft Diagnostic AI Outperforms Generalist… unite.ai[3] 1023 Jack – What the Science paper actually measured 1023jack.com[5] arXiv 2506.22405 – Sequential Diagnosis with Language Models arxiv.org
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- In a text-only, step-by-step simulation built from 304 NEJM case records, MAI-DxO reached 83.2% diagnostic accuracy. [7] [3] [5]
- Original source
- TechCrunch AI article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 22, 2026 article published
- Covered
- Oct 23 edition
- Evidence
- 10 sources across 9 sites
- 1Microsoft’s Mustafa Suleyman says its AI outperforms…techcrunch.com
- 2The Verge – Microsoft says its AI tool is better than…theverge.com
- 31023 Jack – What the Science paper actually measured1023jack.com
- 4Microsoft AI – The Path to Medical Superintelligencemicrosoft.ai
- 5arXiv 2506.22405 – Sequential Diagnosis with Language Modelsarxiv.org
- 6Unite.AI – Google's AMIE study in The Lancetunite.ai
Reach: not measured yet
In a text-only, step-by-step simulation built from 304 NEJM case records, MAI-DxO reached 83.2% diagnostic accuracy. Generalist physicians using reference materials reached 36.1%. On 325 emergency-department encounters from MIMIC-IV-ED, the system scored 80.3% against 46.8% for physicians. [7] [3] [5]
The underlying model mattered more than Microsoft's orchestration layer. GPT-5 alone scored 78.9% and gained about 4 points with MAI-DxO. Orchestration lifted the weaker Gemini 2.5 Flash from 46.4% to about 69%, and it added little on emergency cases with either GPT model. [1]
The cost figures are modelled from a 2023 US hospital price list plus assumed visit fees. Without budget limits, MAI-DxO cost an estimated $5,431 per case, against $4,725 for physicians with references. Budget-limited setups got down to $1,303. One analysis found the system cost more than physicians on emergency cases. [7] [3] [8]
Evidence from real clinical practice is still to come. Microsoft plans a prospective evaluation with Beth Israel Deaconess and is building a healthcare foundation model with Mayo Clinic, which Mayo will own, for release in 2027. Google's AMIE has already published a prospective, physician-supervised study of 98 urgent-care patients in The Lancet. [1] [7] [6]
in simulation, the choice of base model moved accuracy more than orchestration did. Implication: buyers should weigh model selection first.
Read the full assessment
Claims of clinical superiority and modelled cost savings remain unproven until the prospective Beth Israel Deaconess results arrive.
Executive brief
A new peer-reviewed Science paper bears on Microsoft's claim of being ahead of doctors. It finds that most of MAI-DxO's diagnostic gain comes from stronger underlying models, mainly OpenAI's GPT-5 and GPT-5.5, with Microsoft's orchestration adding a smaller increment on top (TechCrunch). The headline result is 83.2% accuracy on 304 difficult NEJM cases, against 36.1% for generalist physicians using reference materials. The comparison is a text-only simulation, and it included no specialists or care teams (1023 Jack). Mustafa Suleyman calls it a step toward "medical superintelligence." Microsoft is now planning a prospective comparison against clinicians at Beth Israel Deaconess.
What changed and event timeline
Preprint claims 85.5% vs 20%
Microsoft introduced SDBench and MAI-DxO. In this version, the system paired with o3 scored 85.5% on NEJM cases, while 21 physicians averaged 20%. The work had not been peer reviewed (;).
GPT-5.5 released
OpenAI's newer model later produced MAI-DxO's best results on complex cases ().
Google's AMIE appears in The Lancet
In a prospective study at Beth Israel Deaconess, 98 urgent-care patients chatted with AMIE while physicians supervised. No conversation had to be stopped under the safety criteria ().
Science paper published
The paper reports 83.2% vs 36.1% on NEJM cases and 80.3% vs 46.8% on emergency-department cases ().
Clinical partnerships announced
Microsoft will evaluate MAI-DxO with Beth Israel Deaconess. It is also training a healthcare foundation model with Mayo Clinic, with release planned for 2027 ().
Capabilities and access
- MAI-DxO is an orchestration layer that works with any underlying model.
- Microsoft tested it with models from OpenAI, Google, Anthropic, Meta and DeepSeek. The best results came from GPT-5 and GPT-5.5.
- It is a research system, not an approved clinical tool. Microsoft says safety validation and regulatory approval would be needed before any deployment (Microsoft AI).
Read the full section
- MAI-DxO is an orchestration layer that works with any underlying model. It runs several AI agents as a virtual physician panel: they question the patient, order tests and debate a diagnosis (TechCrunch).
- Microsoft tested it with models from OpenAI, Google, Anthropic, Meta and DeepSeek. The best results came from GPT-5 and GPT-5.5.
- It is a research system, not an approved clinical tool. Microsoft says safety validation and regulatory approval would be needed before any deployment (Microsoft AI).
Technical analysis for researchers and developers
- SDBench turns 304 NEJM clinicopathological case records into step-by-step encounters.
- The paper adds 325 emergency encounters from MIMIC-IV-ED. Physician data covered 56 cases, with 764 diagnoses made without references and 662 with them (Unite.AI).
- How much the orchestration adds.
Read the full section
- Benchmark design. SDBench turns 304 NEJM clinicopathological case records into step-by-step encounters. Questions and test orders reveal information gradually, and the final diagnosis is checked against the published outcome (arXiv).
- Second test set. The paper adds 325 emergency encounters from MIMIC-IV-ED. Physician data covered 56 cases, with 764 diagnoses made without references and 662 with them (Unite.AI).
- How much the orchestration adds.
- GPT-5 alone: 78.9%. Adding MAI-DxO gained about 4 points.
- GPT-5.5: rose from 72% to about 87% on complex cases.
- Gemini 2.5 Flash: rose from 46.4% to about 69%.
- Emergency cases: little gain with either GPT model.
- The gap between weak and strong models was larger than the orchestration boost (TechCrunch).
- Costs are modelled. They come from a 2023 US hospital price list plus assumed visit fees (1023 Jack).
Claims and evidence
- The accuracy claim (83.2% vs 36.1%) is peer reviewed. All 15 authors work at Microsoft, and no independent replication has been reported (1023 Jack; PBX Science).
- The Verge's "up to four times more accurate" figure seems to compare the AI with physicians working without references.
- Cost. Without budget limits, MAI-DxO cost an estimated $5,431 per case, against $4,725 for physicians with references. Budget-limited configurations got down to $1,303 (Unite.AI).
Read the full section
- The accuracy claim (83.2% vs 36.1%) is peer reviewed. All 15 authors work at Microsoft, and no independent replication has been reported (1023 Jack; PBX Science).
- The Verge's "up to four times more accurate" figure seems to compare the AI with physicians working without references. The 22% figure for that condition is reported in Unite.AI's search summary (The Verge).
- Cost. Without budget limits, MAI-DxO cost an estimated $5,431 per case, against $4,725 for physicians with references. Budget-limited configurations got down to $1,303 (Unite.AI).
- Suleyman's forecasts that healthcare models will improve tenfold in 24–36 months, and that 2027 will bring a coding-style breakthrough for medical records, are predictions. No evidence was offered for either (TechCrunch).
Context and prior work
- The 2025 preprint drew criticism for its artificial test conditions and for being used to back bold claims before peer review (CO/AI).
- Commentators in BMJ Leader argued that "AI vs doctors" is the wrong framing for healthcare's future (BMJ Leader).
- Google's AMIE study went further toward real practice: it was prospective and involved real patients under physician supervision (Unite.AI).
Read the full section
- The 2025 preprint drew criticism for its artificial test conditions and for being used to back bold claims before peer review (CO/AI).
- Commentators in BMJ Leader argued that "AI vs doctors" is the wrong framing for healthcare's future (BMJ Leader).
- Google's AMIE study went further toward real practice: it was prospective and involved real patients under physician supervision (Unite.AI).
Limitations, safety and contested findings
- The authors say the results do not show superiority over specialists, multidisciplinary teams or real-world practice.
- The benchmark leaves out healthy patients and benign diagnoses, so false-alarm rates cannot be measured (1023 Jack).
- Emergency-case costs. One analysis found MAI-DxO's per-case cost was higher than physicians' on emergency cases (PBX Science).
Read the full section
- Scope of the comparison. The authors say the results do not show superiority over specialists, multidisciplinary teams or real-world practice. They also say the simulation amplifies the AI's advantages by setting aside uncertain histories, incomplete records, rapport and multimodal data (Unite.AI).
- False alarms untested. The benchmark leaves out healthy patients and benign diagnoses, so false-alarm rates cannot be measured (1023 Jack).
- Emergency-case costs. One analysis found MAI-DxO's per-case cost was higher than physicians' on emergency cases (PBX Science).
Business and practitioner implications
- Picking the base model moves accuracy more than the orchestration layer does.
- Microsoft sees DxO as an extension of its existing dictation and radiology products, and wants to lead in clinical decision support.
- Mayo Clinic will own the frontier healthcare model it is building with Microsoft (Unite.AI).
Read the full section
- Model choice matters most. Picking the base model moves accuracy more than the orchestration layer does. Orchestration is worth most when the model is cheaper or weaker, and when cases are complex (TechCrunch).
- Microsoft's product direction. Microsoft sees DxO as an extension of its existing dictation and radiology products, and wants to lead in clinical decision support.
- Who owns the model. Mayo Clinic will own the frontier healthcare model it is building with Microsoft (Unite.AI).
- What buyers should do. Wait for the Beth Israel Deaconess results before acting on deployment claims. Treat modelled cost savings as estimates, not measured savings.
The source trail.
Sources (10)
Microsoft’s Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases ‘medical superintelligence’
Article text retrieved; extracted text may omit tables or interactive elements.
techcrunch.com