An Assessment of the Performance of Different Chatbots on Shoulder and Elbow Questions

TL;DR

This study evaluates how well AI chatbots answer shoulder and elbow surgery questions, finding that while they perform reasonably, human experts still outperform them.

Contribution

The study is one of the first to assess AI chatbots specifically for shoulder and elbow surgery education.

Findings

GPT-4o had the highest accuracy at 74%, but all chatbots averaged 60.4%.

Chatbots performed worse on complex topics like nerve injuries and hard questions.

AAOS members outperformed chatbots with 79.4% accuracy.

Abstract

Background/Objectives: The utility of artificial intelligence (AI) in medical education has recently garnered significant interest, with several studies exploring its applications across various educational domains; however, its role in orthopedic education, particularly in shoulder and elbow surgery, remains scarcely studied. This study aims to evaluate the performance of multiple AI models in answering shoulder- and elbow-related questions from the AAOS ResStudy question bank. Methods: A total of 50 shoulder- and elbow-related questions from the AAOS ResStudy question bank were selected for the study. Questions were categorized according to anatomical location, topic, concept, and difficulty. Each question, along with the possible multiple-choice answers, was provided to each chatbot. The performance of each chatbot was recorded and analyzed to identify significant differences between the chatbots’ performances across various categories. Results: The overall average performance of all chatbots was 60.4%. There were significant differences in the performances of different chatbots (p = 0.034): GPT-4o performed best, answering 74% of the questions correctly. AAOS members outperformed all chatbots, with an average accuracy of 79.4%. There were no significant differences in performance between shoulder and elbow questions (p = 0.931). Topic-wise, chatbots did worse on questions relating to “Adhesive Capsulitis” than those relating to “Instability” (p = 0.013), “Nerve Injuries” (p = 0.002), and “Arthroplasty” (p = 0.028). Concept-wise, the best performance was seen in “Diagnosis” (71.4%), but there were no significant differences in scores between different chatbots. Difficulty analysis revealed that chatbots performed significantly better on easy questions (68.5%) compared to moderate (45.4%; p = 0.04) and hard questions (40.0%; p = 0.012). Conclusions: AI chatbots show promise as supplementary tools in medical education and clinical decision-making, but their limitations necessitate cautious and complementary use alongside expert human judgment.