# Harnessing advanced large language models in otolaryngology board examinations: an investigation using python and application programming interfaces

## TL;DR

This study tested how well advanced AI models can answer otolaryngology board exam questions, finding that newer models perform better than older ones.

## Contribution

The study evaluates the performance of multiple advanced LLMs on specialized otolaryngology board questions and tracks changes in GPT-3.5 Turbo over time.

## Key findings

- GPT-4o achieved the highest accuracy, especially in allergology and head and neck tumors.
- GPT-3.5 Turbo's accuracy significantly declined over the past year.
- Single-choice questions yielded higher accuracy than multiple-choice questions across all models.

## Abstract

This study aimed to explore the capabilities of advanced large language models (LLMs), including OpenAI’s GPT-4 variants, Google’s Gemini series, and Anthropic’s Claude series, in addressing highly specialized otolaryngology board examination questions. Additionally, the study included a longitudinal assessment of GPT-3.5 Turbo, which was evaluated using the same set of questions one year ago to identify changes in its performance over time.

We utilized a question bank comprising 2,576 multiple-choice and single-choice questions from a German online education platform tailored for otolaryngology board certification preparation. The questions were submitted to 11 different LLMs, including GPT-3.5 Turbo, GPT-4 variants, Gemini models, and Claude models, through Application Programming Interfaces (APIs) using Python scripts, facilitating efficient data collection and processing.

GPT-4o demonstrated the highest accuracy among all models, particularly excelling in categories such as allergology and head and neck tumors. While the Claude models showed competitive performance, they generally lagged behind the GPT-4 variants. A comparison of GPT-3.5 Turbo’s performance revealed a significant decline in accuracy over the past year. Newer LLMs displayed varied performance levels, with single-choice questions consistently yielding higher accuracy than multiple-choice questions across all models.

While newer LLMs show strong potential in addressing specialized medical content, the observed decline in GPT-3.5 Turbo’s performance over time underscores the necessity for continuous evaluation. This study highlights the critical need for ongoing optimization and efficient API usage to improve LLMs potential for applications in medical education and certification.