Testing for lack of fit in paired comparison data

TL;DR

This paper introduces new statistical tests to detect violations of linear stochastic transitivity in paired comparison data, improving power over classical methods and applicable to high-dimensional, sparse comparison graphs.

Contribution

Develops a suite of tests for lack of fit in paired comparison models, extending to high-dimensional sparse graphs with theoretical analysis and practical validation.

Findings

Tests outperform classical Kendall--Smith in power.

Application reveals substantial intransitivity in real data.

Framework applicable to high-dimensional, sparse comparison graphs.

Abstract

Linear stochastic transitivity is a central assumption in paired comparison models that is rarely verified in practice. Empirical violations, however, are common and can substantially affect inference and ranking. We develop a class of tests for detecting lack of fit in cardinal paired comparison models, where lack of fit is characterized by the presence of cyclical preferences among subsets of items. We propose a suite of tests adapted to different regimes governing the growth of the comparison graph. For a fixed number of items, the proposed procedures exhibit substantially improved power relative to the classical Kendall--Smith test and its cardinal analogue. We further extend the framework to high--dimensional, sparse comparison graphs near the connectivity threshold in random graph models. The theoretical analysis characterizes the behavior of the tests under both the null and alternative, with particular emphasis on limits of detectability and consistency. Simulation studies corroborate the theoretical findings, and applications to real data uncover substantial and previously unrecognized intransitivity and structural lack of fit.