Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback

TL;DR

This paper introduces a mathematically grounded framework for reward modeling using ordinal Likert scale preferences, enabling more effective use of nuanced human feedback in aligning language models.

Contribution

It formulates reward modeling as an ordinal regression problem and derives loss functions that learn thresholds directly from data, improving over heuristic methods.

Findings

Achieves competitive or superior performance on multiple benchmarks.

Effectively captures the ordinal structure of human preferences.

Provides a principled framework for integrating Likert scale feedback.

Abstract

Reward modeling is crucial for aligning large language models with human preferences, yet current approaches lack a principled mathematical framework for leveraging ordinal preference data. When human annotators provide graded preferences on a Likert scale (e.g., significantly better, better, slightly better, negligibly better), existing methods typically apply ad-hoc heuristics, such as margin terms or scaling factors, to loss functions derived from binary preference models like Bradley-Terry. These approaches lack an underlying mathematical model for how ordinal preference data is generated. We present a theoretically grounded framework that formulates reward modeling with Likert scale preferences as a discrete ordinal regression problem. We derive two loss functions from this formulation: a negative log-likelihood loss and an all-threshold loss, both of which learn threshold parameters that naturally capture the ordinal structure of preferences. Unlike existing heuristic methods that manually specify fixed margins or scaling weights, our approach learns these parameters directly from data within a coherent probabilistic framework. Experimental results on multiple benchmarks demonstrate that our ordinal regression approach consistently achieves competitive or superior performance compared to existing heuristic methods across diverse evaluation categories including chat, reasoning, and safety tasks. Our work provides the first principled mathematical framework for incorporating Likert scale preferences into reward model training, moving beyond ad-hoc modifications of binary preference models to enable more effective utilization of fine-grained human feedback.