Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective

TL;DR

This paper demonstrates that leveraging sparsity in preference models can significantly improve sample efficiency and prediction accuracy, especially in high-dimensional settings, through theoretical analysis and empirical validation.

Contribution

It establishes sharp error bounds for sparse preference learning and shows that sparsity-aware methods outperform traditional approaches in sample efficiency.

Findings

Sparsity reduces the minimax error rate to (k/n (d/k)).

The -regularized estimator achieves near-optimal rates under mild conditions.

Experiments confirm that sparsity-aware methods improve accuracy and reduce sample complexity.

Abstract

This paper considers the sample-efficiency of preference learning, which models and predicts human choices based on comparative judgments. The minimax optimal estimation error rate $\Theta(d/n)$ in classical estimation theory requires that the number of samples $n$ scales linearly with the dimensionality of the feature space $d$. However, the high dimensionality of the feature space and the high cost of collecting human-annotated data challenge the efficiency of traditional estimation methods. To remedy this, we leverage sparsity in the preference model and establish sharp error rates. We show that under the sparse random utility model, where the parameter of the reward function is $k$-sparse, the minimax optimal rate can be reduced to $\Theta(k/n \log(d/k))$. Furthermore, we analyze the $\ell_{1}$-regularized estimator and show that it achieves near-optimal rate under mild assumptions on the Gram matrix. Experiments on synthetic data and LLM alignment data validate our theoretical findings, showing that sparsity-aware methods significantly reduce sample complexity and improve prediction accuracy.