Hypothesis Testing for Generalized Thurstone Models

TL;DR

This paper develops a hypothesis testing framework for generalized Thurstone models, providing bounds, tests, and validation methods to determine if pairwise comparison data fits such models.

Contribution

It introduces a minimax hypothesis testing approach for generalized Thurstone models, including bounds, tests, and validation techniques, addressing a fundamental problem in the field.

Findings

Threshold scales as (nk)^{-1/2} for complete graphs

Proposed tests control type I and II errors effectively

Validated results on synthetic and real datasets

Abstract

In this work, we develop a hypothesis testing framework to determine whether pairwise comparison data is generated by an underlying \emph{generalized Thurstone model} $\mathcal{T}_F$ for a given choice function $F$. While prior work has predominantly focused on parameter estimation and uncertainty quantification for such models, we address the fundamental problem of minimax hypothesis testing for $\mathcal{T}_F$ models. We formulate this testing problem by introducing a notion of separation distance between general pairwise comparison models and the class of $\mathcal{T}_F$ models. We then derive upper and lower bounds on the critical threshold for testing that depend on the topology of the observation graph. For the special case of complete observation graphs, this threshold scales as $\Theta((nk)^{-1/2})$, where $n$ is the number of agents and $k$ is the number of comparisons per pair. Furthermore, we propose a hypothesis test based on our separation distance, construct confidence intervals, establish time-uniform bounds on the probabilities of type I and II errors using reverse martingale techniques, and derive minimax lower bounds using information-theoretic methods. Finally, we validate our results through experiments on synthetic and real-world datasets.