Can large language models reason about medical questions?

TL;DR

This study evaluates large language models' ability to reason about complex medical questions, demonstrating that with advanced prompting techniques, models like GPT-3.5 can achieve passing scores on medical benchmarks, with open-source models closing the gap.

Contribution

It provides a comprehensive assessment of LLMs' reasoning in medical domains using various prompting strategies and shows open-source models' potential to match proprietary models.

Findings

GPT-3.5 achieves passing scores on three medical benchmarks.

Prompt engineering significantly improves model performance.

Open-source Llama-2 70B passes MedQA-USMLE with 62.5% accuracy.

Abstract

Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether close- and open-source models (GPT-3.5, LLama-2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-USMLE, MedMCQA, and PubMedQA) and multiple prompting scenarios: Chain-of-Thought (CoT, think step-by-step), few-shot and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions, but also reaches the passing score on three datasets: MedQA-USMLE 60.2%, MedMCQA 62.7% and PubMedQA 78.2%. Open-source models are closing the gap: Llama-2 70B also passed the MedQA-USMLE with 62.5% accuracy.