Generating AI Literacy MCQs: A Multi-Agent LLM Approach

TL;DR

This paper presents a novel multi-agent LLM-based method for automatically generating high-quality AI literacy MCQs tailored to specific educational standards, aiming to enhance K-12 AI education resources.

Contribution

It introduces an iterative LLM-powered workflow with critique agents for pedagogically aligned question generation, addressing the lack of scalable AI literacy assessment tools.

Findings

Experts showed strong interest in the generated MCQs

The system can produce questions aligned with learning objectives and Bloom's levels

Potential to enrich AI literacy educational materials

Abstract

Artificial intelligence (AI) is transforming society, making it crucial to prepare the next generation through AI literacy in K-12 education. However, scalable and reliable AI literacy materials and assessment resources are lacking. To address this gap, our study presents a novel approach to generating multiple-choice questions (MCQs) for AI literacy assessments. Our method utilizes large language models (LLMs) to automatically generate scalable, high-quality assessment questions. These questions align with user-provided learning objectives, grade levels, and Bloom's Taxonomy levels. We introduce an iterative workflow incorporating LLM-powered critique agents to ensure the generated questions meet pedagogical standards. In the preliminary evaluation, experts expressed strong interest in using the LLM-generated MCQs, indicating that this system could enrich existing AI literacy materials and provide a valuable addition to the toolkit of K-12 educators.