Evaluating Large Language Models in Ophthalmology

TL;DR

This study evaluates the performance of three large language models in ophthalmology, finding GPT-4 performs at a level comparable to experienced physicians and could enhance medical education and clinical decision-making.

Contribution

It provides a comparative analysis of LLMs' ophthalmology knowledge against medical professionals, highlighting GPT-4's superior performance.

Findings

GPT-4 performs at the level of attending physicians.

LLMs outperform medical undergraduates.

GPT-4 shows higher stability and confidence.

Abstract

Purpose: The performance of three different large language models (LLMS) (GPT-3.5, GPT-4, and PaLM2) in answering ophthalmology professional questions was evaluated and compared with that of three different professional populations (medical undergraduates, medical masters, and attending physicians). Methods: A 100-item ophthalmology single-choice test was administered to three different LLMs (GPT-3.5, GPT-4, and PaLM2) and three different professional levels (medical undergraduates, medical masters, and attending physicians), respectively. The performance of LLM was comprehensively evaluated and compared with the human group in terms of average score, stability, and confidence. Results: Each LLM outperformed undergraduates in general, with GPT-3.5 and PaLM2 being slightly below the master's level, while GPT-4 showed a level comparable to that of attending physicians. In addition, GPT-4 showed significantly higher answer stability and confidence than GPT-3.5 and PaLM2. Conclusion: Our study shows that LLM represented by GPT-4 performs better in the field of ophthalmology. With further improvements, LLM will bring unexpected benefits in medical education and clinical decision making in the near future.