Evaluation of ChatGPT-Generated Medical Responses: A Systematic Review and Meta-Analysis

TL;DR

This systematic review and meta-analysis assesses ChatGPT's accuracy in medical responses, highlighting its potential in healthcare but also emphasizing the need for standardized evaluation methods and better reporting practices.

Contribution

The paper provides a comprehensive summary and meta-analysis of existing studies on ChatGPT's medical performance, identifying methodological inconsistencies and guiding future research directions.

Findings

ChatGPT has an overall accuracy of 56% in medical queries.

Study heterogeneity and reporting issues limit reliability of results.

Potential for healthcare applications is promising despite current limitations.

Abstract

Large language models such as ChatGPT are increasingly explored in medical domains. However, the absence of standard guidelines for performance evaluation has led to methodological inconsistencies. This study aims to summarize the available evidence on evaluating ChatGPT's performance in medicine and provide direction for future research. We searched ten medical literature databases on June 15, 2023, using the keyword "ChatGPT". A total of 3520 articles were identified, of which 60 were reviewed and summarized in this paper and 17 were included in the meta-analysis. The analysis showed that ChatGPT displayed an overall integrated accuracy of 56% (95% CI: 51%-60%, I2 = 87%) in addressing medical queries. However, the studies varied in question resource, question-asking process, and evaluation metrics. Moreover, many studies failed to report methodological details, including the version of ChatGPT and whether each question was used independently or repeatedly. Our findings revealed that although ChatGPT demonstrated considerable potential for application in healthcare, the heterogeneity of the studies and insufficient reporting may affect the reliability of these results. Further well-designed studies with comprehensive and transparent reporting are needed to evaluate ChatGPT's performance in medicine.