Statistical Models of Top-$k$ Partial Orders

TL;DR

This paper introduces new statistical models for jointly analyzing top-$k$ partial orders and list lengths, improving the realism of synthetic data and prediction accuracy in preference ranking systems.

Contribution

It develops and compares composite and augmented models for joint distribution over partial orders and list lengths, filling a gap in existing ranking models.

Findings

Composite models accurately reproduce length distributions in synthetic data.

Augmented models with position-dependent utilities best predict observed data.

Models improve simulation and evaluation of preference-based social systems.

Abstract

In many contexts involving ranked preferences, agents submit partial orders over available alternatives. Statistical models often treat these as marginal in the space of total orders, but this approach overlooks information contained in the list length itself. In this work, we introduce and taxonomize approaches for jointly modeling distributions over top-$k$ partial orders and list lengths $k$, considering two classes of approaches: composite models that view a partial order as a truncation of a total order, and augmented ranking models that model the construction of the list as a sequence of choice decisions, including the decision to stop. For composite models, we consider three dependency structures for joint modeling of order and truncation length. For augmented ranking models, we consider different assumptions on how the stop-token choice is modeled. Using data consisting of partial rankings from San Francisco school choice and San Francisco ranked choice elections, we evaluate how well the models predict observed data and generate realistic synthetic datasets. We find that composite models, explicitly modeling length as a categorical variable, produce synthetic datasets with accurate length distributions, and an augmented model with position-dependent item utilities jointly models length and preferences in the training data best, as measured by negative log loss. Methods from this work have significant implications on the simulation and evaluation of real-world social systems that solicit ranked preferences.