Ordinal Regression for Difficulty Estimation of StepMania Levels

TL;DR

This paper formalizes and evaluates ordinal regression models for predicting the difficulty of StepMania levels, demonstrating neural networks' superiority and validating models through user experiments.

Contribution

It introduces a standardized, extensive dataset for difficulty prediction and shows neural network models outperform existing methods in this domain.

Findings

Neural network models significantly outperform traditional methods.

Extended datasets improve model training and evaluation.

Models are validated through user experiments showing superiority over human labels.

Abstract

StepMania is a popular open-source clone of a rhythm-based video game. As is common in popular games, there is a large number of community-designed levels. It is often difficult for players and level authors to determine the difficulty level of such community contributions. In this work, we formalize and analyze the difficulty prediction task on StepMania levels as an ordinal regression (OR) task. We standardize a more extensive and diverse selection of this data resulting in five data sets, two of which are extensions of previous work. We evaluate many competitive OR and non-OR models, demonstrating that neural network-based models significantly outperform the state of the art and that StepMania-level data makes for an excellent test bed for deep OR models. We conclude with a user experiment showing our trained models' superiority over human labeling.