OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology

TL;DR

OphthBench is a comprehensive benchmark designed to evaluate large language models in Chinese ophthalmology across clinical scenarios, highlighting current limitations and guiding future improvements for practical medical applications.

Contribution

The paper introduces OphthBench, a specialized benchmark with 9 tasks and 591 questions, to systematically assess LLM performance in Chinese ophthalmic clinical workflows.

Findings

39 LLMs evaluated, revealing performance gaps

Benchmark covers education, triage, diagnosis, treatment, prognosis

Provides insights for future LLM development in ophthalmology

Abstract

Large language models (LLMs) have shown significant promise across various medical applications, with ophthalmology being a notable area of focus. Many ophthalmic tasks have shown substantial improvement through the integration of LLMs. However, before these models can be widely adopted in clinical practice, evaluating their capabilities and identifying their limitations is crucial. To address this research gap and support the real-world application of LLMs, we introduce the OphthBench, a specialized benchmark designed to assess LLM performance within the context of Chinese ophthalmic practices. This benchmark systematically divides a typical ophthalmic clinical workflow into five key scenarios: Education, Triage, Diagnosis, Treatment, and Prognosis. For each scenario, we developed multiple tasks featuring diverse question types, resulting in a comprehensive benchmark comprising 9 tasks and 591 questions. This comprehensive framework allows for a thorough assessment of LLMs' capabilities and provides insights into their practical application in Chinese ophthalmology. Using this benchmark, we conducted extensive experiments and analyzed the results from 39 popular LLMs. Our evaluation highlights the current gap between LLM development and its practical utility in clinical settings, providing a clear direction for future advancements. By bridging this gap, we aim to unlock the potential of LLMs and advance their development in ophthalmology.