GenAI and Faculty Collaboration Support Feasible Development of Curriculum-Aligned Open-Access Board-Style Questions

TL;DR

A generative AI tool helped create medical exam questions, with most being usable after expert review and showing potential for improving curriculum alignment.

Contribution

A new AI system for generating curriculum-aligned medical exam questions using open-access resources, with expert evaluation and ethical design considerations.

Findings

83% of AI-generated questions were deemed potentially usable with minimal or no changes.

Sentiment analysis showed mild positive feedback from experts on AI-generated items.

The AI revealed mismatches between faculty expectations and learning objectives.

Abstract

A curriculum-aligned generative AI chatbot was developed to create USMLE-style anatomy multiple-choice questions using open-access resources. Subject matter experts evaluated 100 of these questions and 83% were found to be potentially usable, with many needing minimal or no modifications. Sentiment analysis of subject matter expert interviews showed mild positive sentiment while acknowledging both advantages and limitations of AI-generated assessment items. Interestingly, the chatbot revealed discrepancies between faculty expectations and documented learning objectives, highlighting opportunities for curricular improvement. This ethically designed system provides a scalable, cost-effective approach to enhance equity and support local assessment development in medical education.