MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering

TL;DR

This paper introduces MedPAIR, a dataset for evaluating how well large language models and physicians align in identifying relevant information in medical questions, revealing misalignments and improving accuracy when irrelevant content is filtered out.

Contribution

The study presents MedPAIR, a new dataset with relevance annotations from physicians, and compares relevance estimation between physicians and LLMs in medical QA.

Findings

LLMs often do not align with physician relevance judgments.

Filtering irrelevant sentences improves accuracy for both LLMs and physicians.

Relevance estimation impacts downstream medical QA performance.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance on various medical question-answering (QA) benchmarks, including standardized medical exams. However, correct answers alone do not ensure correct logic, and models may reach accurate conclusions through flawed processes. In this study, we introduce the MedPAIR (Medical Dataset Comparing Physicians and AI Relevance Estimation and Question Answering) dataset to evaluate how physician trainees and LLMs prioritize relevant information when answering QA questions. We obtain annotations on 1,300 QA pairs from 36 physician trainees, labeling each sentence within the question components for relevance. We compare these relevance estimates to those for LLMs, and further evaluate the impact of these "relevant" subsets on downstream task performance for both physician trainees and LLMs. We find that LLMs are frequently not aligned with the content relevance estimates of physician trainees. After filtering out physician trainee-labeled irrelevant sentences, accuracy improves for both the trainees and the LLMs. All LLM and physician trainee-labeled data are available at: http://medpair.csail.mit.edu/.