Ordinality in Discrete-level Question Difficulty Estimation: Introducing Balanced DRPS and OrderedLogitNN

TL;DR

This paper introduces a new evaluation metric and a neural network model for question difficulty estimation that better respects the ordinal nature of difficulty levels and addresses class imbalance, improving performance on complex tasks.

Contribution

The study proposes the balanced DRPS metric and OrderedLogitNN model, advancing ordinal regression methods for question difficulty estimation with improved evaluation and performance.

Findings

OrderedLogitNN outperforms other models on complex datasets.

Balanced DRPS effectively captures ordinality and class imbalance.

Neural network extension of ordered logit improves difficulty estimation.

Abstract

Recent years have seen growing interest in Question Difficulty Estimation (QDE) using natural language processing techniques. Question difficulty is often represented using discrete levels, framing the task as ordinal regression due to the inherent ordering from easiest to hardest. However, the literature has neglected the ordinal nature of the task, relying on classification or discretized regression models, with specialized ordinal regression methods remaining unexplored. Furthermore, evaluation metrics are tightly coupled to the modeling paradigm, hindering cross-study comparability. While some metrics fail to account for the ordinal structure of difficulty levels, none adequately address class imbalance, resulting in biased performance assessments. This study addresses these limitations by benchmarking three types of model outputs -- discretized regression, classification, and ordinal regression -- using the balanced Discrete Ranked Probability Score (DRPS), a novel metric that jointly captures ordinality and class imbalance. In addition to using popular ordinal regression methods, we propose OrderedLogitNN, extending the ordered logit model from econometrics to neural networks. We fine-tune BERT on the RACE++ and ARC datasets and find that OrderedLogitNN performs considerably better on complex tasks. The balanced DRPS offers a robust and fair evaluation metric for discrete-level QDE, providing a principled foundation for future research.