# Artificial intelligence vs. human evaluation of anesthesia education videos: a comparative analysis using validated quality scales

## TL;DR

This study compares human and AI evaluations of anesthesia education videos, finding that AI models like ChatGPT-5 can closely match human ratings, though human videos score higher in quality.

## Contribution

The study introduces AI-based evaluation as a scalable alternative to human assessment of medical education videos, with strong correlations between AI and human ratings.

## Key findings

- Human-generated videos scored higher in DISCERN and JAMA quality metrics compared to AI-generated videos.
- ChatGPT-5 showed strong agreement with human ratings across all evaluation scales.
- AI-generated content matched human videos in structural organization and linguistic fluency.

## Abstract

YouTube has become an increasingly popular platform for medical education, yet the accuracy and educational quality of anesthesia-related videos remain uncertain. While human experts have traditionally assessed video quality using validated scales such as DISCERN, JAMA, and the Global Quality Scale (GQS), artificial intelligence (AI) models—particularly large language models (LLMs)—now offer new possibilities for scalable, objective content evaluation. This study aimed to compare the educational quality of anesthesia education videos produced by humans and AI, and to examine the level of agreement between human expert ratings and ChatGPT-5 evaluations.

In this cross-sectional analytical study, forty YouTube videos were analyzed: 20 produced by human educators and 20 generated using AI tools. Each video was independently assessed by two anesthesiologists and by ChatGPT-5 Plus (OpenAI, 2025) using DISCERN, JAMA, and GQS criteria. Inter-rater reliability between human evaluators was determined using the Intraclass Correlation Coefficient (ICC), and correlations between human and AI ratings were analyzed with Spearman’s rho.

Human-generated videos scored significantly higher than AI-generated ones in DISCERN (68.45 ± 4.60 vs. 62.77 ± 7.32, p = 0.0044, Cohen’s d = 0.82) and JAMA (3.70 ± 0.41 vs. 3.23 ± 0.77, p = 0.0446, Cohen’s d = 0.71) scores, whereas no significant difference was observed in GQS scores (p = 0.3033). Inter-rater reliability between human experts was excellent (ICC = 0.81–0.86, p < 0.001). Strong correlations were found between ChatGPT-5 and the human mean scores for all scales (ρ = 0.897 for DISCERN, ρ = 0.785 for GQS, ρ = 0.765 for JAMA; p < 0.001), indicating high agreement between AI and human evaluations.

AI-based models such as ChatGPT-5 show potential to approximate human expert judgment in evaluating educational content. While human-generated videos remain superior in terms of source transparency and ethical reporting, AI-generated content approaches human quality in structural organization and linguistic fluency. These findings suggest that AI-assisted evaluation systems may serve as standardized, efficient tools for quality screening of large-scale educational video archives in medical education.