Large Language Model Benchmarks in Medical Tasks

TL;DR

This paper surveys various benchmark datasets used to evaluate large language models in medical tasks, covering multiple modalities and highlighting their role in advancing clinical AI applications.

Contribution

It provides a comprehensive categorization and analysis of medical benchmark datasets, discussing their significance, challenges, and future opportunities for multimodal medical AI development.

Findings

Key benchmarks include MIMIC-III, MIMIC-IV, BioASQ, PubMedQA, CheXpert

Benchmarks have facilitated progress in report generation and clinical summarization

Identifies challenges like language diversity and data synthesis in medical benchmarks

Abstract

With the increasing application of large language models (LLMs) in the medical domain, evaluating these models' performance using benchmark datasets has become crucial. This paper presents a comprehensive survey of various benchmark datasets employed in medical LLM tasks. These datasets span multiple modalities including text, image, and multimodal benchmarks, focusing on different aspects of medical knowledge such as electronic health records (EHRs), doctor-patient dialogues, medical question-answering, and medical image captioning. The survey categorizes the datasets by modality, discussing their significance, data structure, and impact on the development of LLMs for clinical tasks such as diagnosis, report generation, and predictive decision support. Key benchmarks include MIMIC-III, MIMIC-IV, BioASQ, PubMedQA, and CheXpert, which have facilitated advancements in tasks like medical report generation, clinical summarization, and synthetic data generation. The paper summarizes the challenges and opportunities in leveraging these benchmarks for advancing multimodal medical intelligence, emphasizing the need for datasets with a greater degree of language diversity, structured omics data, and innovative approaches to synthesis. This work also provides a foundation for future research in the application of LLMs in medicine, contributing to the evolving field of medical artificial intelligence.