Appraisal of ChatGPT's responses to common patient questions regarding acromioclavicular joint dislocations

TL;DR

This study evaluates ChatGPT's ability to answer patient questions about AC joint dislocations, finding it generally accurate but with limitations in treatment advice.

Contribution

The study introduces a novel evaluation of ChatGPT's medical responses specifically for AC joint dislocations using surgeon ratings and readability metrics.

Findings

ChatGPT's responses averaged a 4/5 quality score with minor inaccuracies.

Treatment-related questions scored lower than rehabilitation or diagnosis questions.

Readability was suitable for high school-level audiences.

Abstract

ChatGPT, an expanding artificial intelligence platform, is rapidly becoming a source of medical knowledge for patients. The purpose of this study is to evaluate the quality and readability of information provided by ChatGPT 4.0 in response to the most frequently asked patients' questions regarding acromioclavicular (AC) joint dislocations.

Twenty-five frequently asked patient questions regarding AC joint dislocations were posed to ChatGPT 4.0. The quality and accuracy of the responses was graded by two fellowship-trained shoulder surgeons using a 5-point Likert scale. Responses were analyzed for readability using six established metrics.

ChatGPT provided responses to 25 frequently asked questions, with consensus reached on 12 questions. The final average score was 4/5, reflecting good quality with minor inaccuracies. Treatment-related questions scored lower compared to other categories, with a significant difference observed between treatment and rehabilitation (P = .025). Cohen's kappa indicated poor correlation (0.085), though the percent agreement was 48%, increasing to 100% when allowing for a 1-point difference. Readability scores revealed moderate difficulty levels, suitable for a high school-level audience.

ChatGPT delivers accurate and easily comprehensible information on AC joint dislocations, highlighting its potential to improve patient education. Although the model generally provides high-quality responses, its limitations in addressing treatment-related questions underscore the importance of clinician oversight. ChatGPT can therefore serve as a valuable complement to traditional patient education methods.

Figures