Active Learning to Guide Labeling Efforts for Question Difficulty Estimation

TL;DR

This paper introduces an active learning approach with a novel acquisition function for question difficulty estimation, significantly reducing labeling efforts while maintaining high performance.

Contribution

It proposes PowerVariance, a new acquisition function for active learning in QDE, and demonstrates its effectiveness with less labeled data needed.

Findings

Active learning with PowerVariance achieves near state-of-the-art performance using only 10% of labeled data.

The method reduces labeling efforts significantly compared to fully supervised models.

The approach is applicable to educational and personalized question-answering systems.

Abstract

In recent years, there has been a surge in research on Question Difficulty Estimation (QDE) using natural language processing techniques. Transformer-based neural networks achieve state-of-the-art performance, primarily through supervised methods but with an isolated study in unsupervised learning. While supervised methods focus on predictive performance, they require abundant labeled data. On the other hand, unsupervised methods do not require labeled data but rely on a different evaluation metric that is also computationally expensive in practice. This work bridges the research gap by exploring active learning for QDE, a supervised human-in-the-loop approach striving to minimize the labeling efforts while matching the performance of state-of-the-art models. The active learning process iteratively trains on a labeled subset, acquiring labels from human experts only for the most informative unlabeled data points. Furthermore, we propose a novel acquisition function PowerVariance to add the most informative samples to the labeled set, a regression extension to the PowerBALD function popular in classification. We employ DistilBERT for QDE and identify informative samples by applying Monte Carlo dropout to capture epistemic uncertainty in unlabeled samples. The experiments demonstrate that active learning with PowerVariance acquisition achieves a performance close to fully supervised models after labeling only 10% of the training data. The proposed methodology promotes the responsible use of educational resources, makes QDE tools more accessible to course instructors, and is promising for other applications such as personalized support systems and question-answering tools.