VizQStudio: Iterative Visualization Literacy MCQs Design with Simulated Students

TL;DR

VizQStudio is a visual analytics tool that leverages MLLM-powered simulated students to assist instructors in designing, refining, and understanding visualization literacy MCQs through iterative, exploratory processes before classroom use.

Contribution

The paper introduces VizQStudio, a novel system that enables iterative visualization literacy MCQ design using simulated students, addressing limitations of fixed item banks and supporting instructor-centered assessment creation.

Findings

Effective exploration of question design trade-offs

Identification of misconceptions and difficulty calibration

Insights into AI-assisted assessment design limitations

Abstract

Multiple-choice questions (MCQs) are a widely used educational tool, particularly in domains such as visualization literacy that require broad conceptual coverage and support diverse real-world applications. However, designing high-quality visualization literacy MCQs remains challenging, as instructors must coordinate multimodal elements (e.g., charts, question stems, and distractors), address diverse visualization tasks, and accommodate learners with heterogeneous backgrounds. Existing visualization literacy assessments primarily rely on standardized, fixed item banks, offering limited support for iterative question design that adapts to differences in learners' abilities, backgrounds, and reasoning strategies. To address these challenges, we present VizQStudio, a visual analytics system that supports instructors in iteratively designing and refining visualization literacy MCQs using MLLM-powered simulated students. Instructors can specify diverse student profiles spanning demographics, knowledge levels, and learning-related traits. The system then visualizes how simulated students reason about and respond to different question components, helping instructors explore potential misconceptions, difficulty calibration, and design trade-offs prior to classroom deployment. We investigate VizQStudio through a mixed-method evaluation, including expert interviews, case studies, a classroom deployment, and a large-scale online study. Overall, this work reframes MLLM-based student simulation in assessment authoring as a design-time, exploratory aid. By examining both its value and limitations in realistic instructional settings, we surface design insights that inform how future systems can support instructor-centered, iterative, and responsible uses of AI for multimodal assessment design in visualization literacy and related domains.