Optimizing Coverage and Difficulty in Reinforcement Learning for Quiz Composition

TL;DR

This paper proposes a reinforcement learning framework to automate quiz composition, optimizing for topic coverage and difficulty, demonstrated through extensive experiments and a user study.

Contribution

It formalizes quiz composition as a sequential decision-making problem and evaluates multiple RL algorithms, highlighting their behaviors and transfer learning capabilities.

Findings

RL agents can effectively optimize quiz quality based on coverage and difficulty.

Different RL algorithms exhibit subtle behavioral differences and transfer learning performance.

User study supports the potential of RL for automating pedagogical goals.

Abstract

Quiz design is a tedious process that teachers undertake to evaluate the acquisition of knowledge by students. Our goal in this paper is to automate quiz composition from a set of multiple choice questions (MCQs). We formalize a generic sequential decision-making problem with the goal of training an agent to compose a quiz that meets the desired topic coverage and difficulty levels. We investigate DQN, SARSA and A2C/A3C, three reinforcement learning solutions to solve our problem. We run extensive experiments on synthetic and real datasets that study the ability of RL to land on the best quiz. Our results reveal subtle differences in agent behavior and in transfer learning with different data distributions and teacher goals. This was supported by our user study, paving the way for automating various teachers' pedagogical goals.