From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns

TL;DR

This study evaluates the quality of AI-generated multiple-choice questions for Portuguese reading comprehension, highlighting strengths and challenges in semantic clarity, distractor quality, and real-world applicability for elementary education.

Contribution

It provides an empirical assessment of generative AI models for Portuguese MCQ creation, focusing on narrative alignment, difficulty levels, and psychometric evaluation.

Findings

AI models produce MCQs comparable to human-authored ones.

Semantic clarity and distractor quality remain challenging.

Generated MCQs show potential for educational use with improvements.

Abstract

While MCQs are valuable for learning and evaluation, manually creating them with varying difficulty levels and targeted reading skills remains a time-consuming and costly task. Recent advances in generative AI provide an opportunity to automate MCQ generation efficiently. However, assessing the actual quality and reliability of generated MCQs has received limited attention -- particularly regarding cases where generation fails. This aspect becomes particularly important when the generated MCQs are meant to be applied in real-world settings. Additionally, most MCQ generation studies focus on English, leaving other languages underexplored. This paper investigates the capabilities of current generative models in producing MCQs for reading comprehension in Portuguese, a morphologically rich language. Our study focuses on generating MCQs that align with curriculum-relevant narrative elements and span different difficulty levels. We evaluate these MCQs through expert review and by analyzing the psychometric properties extracted from student responses to assess their suitability for elementary school students. Our results show that current models can generate MCQs of comparable quality to human-authored ones. However, we identify issues related to semantic clarity and answerability. Also, challenges remain in generating distractors that engage students and meet established criteria for high-quality MCQ option design.