Preference Models assume Proportional Hazards of Utilities

TL;DR

This paper links the Plackett-Luce preference model to the Cox Proportional Hazards model, providing new insights into preference estimation methods used in AI alignment tools.

Contribution

It establishes a novel connection between preference modeling and survival analysis, illuminating underlying assumptions in current AI preference estimation techniques.

Findings

Connects Plackett-Luce to Cox Proportional Hazards model

Provides insights into preference estimation assumptions

Suggests implications for AI alignment methods

Abstract

Approaches for estimating preferences from human annotated data typically involves inducing a distribution over a ranked list of choices such as the Plackett-Luce model. Indeed, modern AI alignment tools such as Reward Modelling and Direct Preference Optimization are based on the statistical assumptions posed by the Plackett-Luce model. In this paper, I will connect the Plackett-Luce model to another classical and well known statistical model, the Cox Proportional Hazards model and attempt to shed some light on the implications of the connection therein.