Response Time Enhances Alignment with Heterogeneous Preferences

TL;DR

This paper introduces a method that uses response time data, modeled with Drift-Diffusion Models, to accurately identify population preferences in LLM alignment, overcoming limitations of choice-only data.

Contribution

It proposes augmenting preference datasets with response times and models them with DDMs to restore preference identifiability and correct biases in heterogeneous labeler populations.

Findings

Response time data improves preference estimation accuracy.

The proposed estimator converges to true preferences even with minimal labeler data.

Empirical results outperform standard choice-only baselines.

Abstract

Aligning large language models (LLMs) to human preferences typically relies on aggregating pooled feedback into a single reward model. However, this standard approach assumes that all labelers share the same underlying preferences, ignoring the fact that real-world labelers are highly heterogeneous and usually anonymous. Consequently, relying solely on binary choice data fundamentally distorts the learned policy, making the true population-average preference unidentifiable. To overcome this critical limitation, we demonstrate that augmenting preference datasets with a simple, secondary signal -- the user's response time -- can restore the identifiability of the population's average preference. By modeling each decision as a Drift-Diffusion Model (DDM), we introduce a novel, consistent estimator of heterogeneous preferences that successfully corrects the distortions of standard choice-only labels. We prove that our estimator asymptotically converges to the true average preference even in extreme cases where each anonymous labeler contributes only a single choice. Empirically, across both synthetic and real-world datasets, our method consistently outperforms standard baselines that otherwise fail and plateau at a bias floor. Because response times are essentially free to record and require zero user tracking or identification, our results bring promises and open up new opportunities for future data-collection pipelines to improve the social benefit without requiring user-level identifiers or repeated elicitations.