Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks

TL;DR

This paper introduces RABBITS, a robustness dataset revealing that language models' performance on biomedical benchmarks significantly drops when drug names are swapped between brand and generic terms, highlighting their fragility.

Contribution

The paper creates RABBITS, a new dataset to evaluate LLM robustness to drug name variations, and demonstrates performance drops and potential data contamination issues in biomedical NLP models.

Findings

Performance drops of 1-10% after drug name swapping.

Test data contamination in pre-training datasets.

Open-source and API-based LLMs show similar fragility.

Abstract

Medical knowledge is context-dependent and requires consistent reasoning across various natural language expressions of semantically equivalent phrases. This is particularly crucial for drug names, where patients often use brand names like Advil or Tylenol instead of their generic equivalents. To study this, we create a new robustness dataset, RABBITS, to evaluate performance differences on medical benchmarks after swapping brand and generic drug names using physician expert annotations. We assess both open-source and API-based LLMs on MedQA and MedMCQA, revealing a consistent performance drop ranging from 1-10\%. Furthermore, we identify a potential source of this fragility as the contamination of test data in widely used pre-training datasets. All code is accessible at https://github.com/BittermanLab/RABBITS, and a HuggingFace leaderboard is available at https://huggingface.co/spaces/AIM-Harvard/rabbits-leaderboard.