Diagnostic Assessment Generation via Combinatorial Search

TL;DR

This paper introduces a genetic algorithm-based method for automatically generating diagnostic assessment tests from learner problem-solving data, improving test quality and efficiency.

Contribution

It presents a novel combinatorial search formulation and a genetic algorithm approach for assembling effective diagnostic tests from raw learner data.

Findings

Outperforms greedy and random baselines significantly.

Produces tests with good problem distribution and difficulty balance.

Effective across multiple datasets.

Abstract

Initial assessment tests are crucial in capturing learner knowledge states in a consistent manner. Aside from crafting questions itself, putting together relevant problems to form a question sheet is also a time-consuming process. In this work, we present a generic formulation of question assembly and a genetic algorithm based method that can generate assessment tests from raw problem-solving history. First, we estimate the learner-question knowledge matrix (snapshot). Each matrix element stands for the probability that a learner correctly answers a specific question. We formulate the task as a combinatorial search over this snapshot. To ensure representative and discriminative diagnostic tests, questions are selected (1) that has a low root mean squared error against the whole question pool and (2) high standard deviation among learner performances. Experimental results show that the proposed method outperforms greedy and random baseline by a large margin in one private dataset and four public datasets. We also performed qualitative analysis on the generated assessment test for 9th graders, which enjoys good problem scatterness across the whole 9th grader curriculum and decent difficulty level distribution.