MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology

TL;DR

MTBBench is a new benchmark simulating complex, multimodal, and longitudinal decision-making in oncology, revealing current LLMs' limitations and guiding improvements for clinical reasoning tasks.

Contribution

Introduces MTBBench, a realistic, multimodal, and longitudinal benchmark for oncology decision-making, along with an agentic framework that improves LLM reasoning and reliability.

Findings

Current LLMs often hallucinate and struggle with multimodal data.

Tool-enhanced models show performance improvements of up to 11.2%.

MTBBench provides a challenging environment for advancing clinical reasoning in LLMs.

Abstract

Multimodal Large Language Models (LLMs) hold promise for biomedical reasoning, but current benchmarks fail to capture the complexity of real-world clinical workflows. Existing evaluations primarily assess unimodal, decontextualized question-answering, overlooking multi-agent decision-making environments such as Molecular Tumor Boards (MTBs). MTBs bring together diverse experts in oncology, where diagnostic and prognostic tasks require integrating heterogeneous data and evolving insights over time. Current benchmarks lack this longitudinal and multimodal complexity. We introduce MTBBench, an agentic benchmark simulating MTB-style decision-making through clinically challenging, multimodal, and longitudinal oncology questions. Ground truth annotations are validated by clinicians via a co-developed app, ensuring clinical relevance. We benchmark multiple open and closed-source LLMs and show that, even at scale, they lack reliability -- frequently hallucinating, struggling with reasoning from time-resolved data, and failing to reconcile conflicting evidence or different modalities. To address these limitations, MTBBench goes beyond benchmarking by providing an agentic framework with foundation model-based tools that enhance multi-modal and longitudinal reasoning, leading to task-level performance gains of up to 9.0% and 11.2%, respectively. Overall, MTBBench offers a challenging and realistic testbed for advancing multimodal LLM reasoning, reliability, and tool-use with a focus on MTB environments in precision oncology.