# Integrating ChatGPT into knowledge-retrieval tutorials in undergraduate medical education: a prospective evaluation of higher-order learning and feasibility

## TL;DR

This study shows that using ChatGPT in medical education improves students' higher-order thinking skills and learning retention over time.

## Contribution

The first longitudinal study to evaluate ChatGPT-assisted retrieval practice in medical education, showing durable learning gains.

## Key findings

- Students showed significant and sustained improvement in higher-order MCQ scores after ChatGPT-assisted tutorials.
- Lower-achieving students benefited more from the intervention compared to higher-achieving peers.
- Student satisfaction and engagement were high, though clarity of AI interaction was rated lower.

## Abstract

Generative artificial intelligence (AI) platforms such as ChatGPT present new opportunities to strengthen competency-based medical education (CBME). While retrieval practice is a proven strategy for enhancing long-term retention, its application to higher-order domains of Bloom’s taxonomy within CBME, particularly when scaffolded by AI, remains underexplored. To our knowledge, this is the first longitudinal CBME study worldwide to evaluate supervised ChatGPT-assisted retrieval practice and its durability over time.

We conducted a six-month, prospective, non-randomised delayed-intervention study and all 270 third-year MBBS students (including supplementary batch) at a private medical college in South India were invited to participate in a faculty-supervised ChatGPT-assisted retrieval-practice intervention. Participants were allocated into four tutorial clusters; two received the intervention immediately, while the remaining two received it after a delay. The intervention comprised four weekly, two-hour sessions featuring higher-order multiple-choice questions, structured faculty-supervised interactions with ChatGPT, and guided metacognitive reflections. Outcomes assessed included MCQ performance at baseline, immediately post-intervention, and at one- and three-month follow-up, as well as student perceptions. Data analysis employed repeated-measures ANOVA and mixed-effects modelling.

Of 270 students invited, 253 (93.7 %) met inclusion criteria and completed all assessments, achieving 100% follow-up at immediate, one-month, and three-month evaluations. MCQ performance demonstrated significant improvement across time points (F[3,756] = 65.14, p < 0.001). On a 20-point higher-order MCQ scale, mean scores increased from 12.8 ± 2.4 at baseline to 15.2 ± 2.1 immediately post-intervention (adjusted gain +2.37, d = 0.62, p < 0.001). Gains were sustained at one month (+3.78, d = 0.99) and three months (+3.65, d = 0.95), demonstrating durable higher-order learning retention. Both Lower-achieving and Higher-achieving students improved, though the effect was greater among Lower-achieving students (d = 0.72 vs 0.49). Student feedback revealed high levels of satisfaction (mean 4.25 ± 0.88) and cognitive engagement (4.15 ± 0.92), while clarity of AI interaction received comparatively lower ratings (3.39 ± 1.19).

Supervised ChatGPT-assisted retrieval practice produced sustained improvements in higher-order cognitive performance, with particularly strong benefits for Lower-achieving students. This scalable, standards-aligned model holds promise for advancing CBME globally and warrants further validation through multi-institutional trials incorporating performance-based and clinical outcomes.