Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solving

TL;DR

Advances in large language models have enhanced their ability to perform flexible and human-like reasoning in medical problem-solving, surpassing previous limitations and reducing reliance on heuristics.

Contribution

This study demonstrates that recent improvements in LLM reasoning capabilities enable greater cognitive flexibility in clinical reasoning tasks, matching human performance on a challenging medical benchmark.

Findings

Strong reasoning LLMs outperform weaker models on medical QA.

Top models answer 55-70% of questions correctly, even on difficult cases.

Models show reduced susceptibility to heuristic traps compared to humans.

Abstract

Large Language Models (LLMs) have achieved high accuracy on medical question-answer (QA) benchmarks, yet their capacity for flexible clinical reasoning has been debated. Here, we asked whether advances in reasoning LLMs improve their cognitive flexibility in clinical reasoning. We assessed reasoning models from the OpenAI, Grok, Gemini, Claude, and DeepSeek families on the medicine abstraction and reasoning corpus (mARC), an adversarial medical QA benchmark which utilizes the Einstellung effect to induce inflexible overreliance on learned heuristic patterns in contexts where they become suboptimal. We found that strong reasoning models avoided Einstellung-based traps more often than weaker reasoning models, achieving human-level performance on mARC. On questions most commonly missed by physicians, the top 5 performing models answered 55% to 70% correctly with high confidence, indicating that these models may be less susceptible than humans to Einstellung effects. Our results indicate that strong reasoning models demonstrate improved flexibility in medical reasoning, achieving performance on par with humans on mARC.