How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset

TL;DR

This paper introduces a dataset to evaluate large language models' understanding of drug mechanisms, testing their factual knowledge and reasoning abilities in novel, counterfactual scenarios relevant to drug development and personalized medicine.

Contribution

The authors present a new dataset for assessing LLMs' knowledge and reasoning about drug mechanisms, and benchmark several models, revealing insights into their strengths and limitations in scientific reasoning.

Findings

o4-mini outperforms several OpenAI models

Qwen3-4B-thinking matches or exceeds o4-mini in some tasks

Reasoning with counterfactuals affecting internal links is more challenging

Abstract

Two scientific fields showing increasing interest in pre-trained large language models (LLMs) are drug development / repurposing, and personalized medicine. For both, LLMs have to demonstrate factual knowledge as well as a deep understanding of drug mechanisms, so they can recall and reason about relevant knowledge in novel situations. Drug mechanisms of action are described as a series of interactions between biomedical entities, which interlink into one or more chains directed from the drug to the targeted disease. Composing the effects of the interactions in a candidate chain leads to an inference about whether the drug might be useful or not for that disease. We introduce a dataset that evaluates LLMs on both factual knowledge of known mechanisms, and their ability to reason about them under novel situations, presented as counterfactuals that the models are unlikely to have seen during training. Using this dataset, we show that o4-mini outperforms the 4o, o3, and o3-mini models from OpenAI, and the recent small Qwen3-4B-thinking model closely matches o4-mini's performance, even outperforming it in some cases. We demonstrate that the open world setting for reasoning tasks, which requires the model to recall relevant knowledge, is more challenging than the closed world setting where the needed factual knowledge is provided. We also show that counterfactuals affecting internal links in the reasoning chain present a much harder task than those affecting a link from the drug mentioned in the prompt.