Preference-based optimization from noisy pairwise comparisons

TL;DR

This paper introduces a preference-based optimization algorithm that learns from noisy pairwise comparisons, effectively converging to a stationary point and demonstrating promising results in control system experiments.

Contribution

It proposes a novel optimization method that utilizes noisy preference feedback and uniform-sphere perturbations, advancing learning from comparison-based data.

Findings

Algorithm converges to a stationary point under standard assumptions

Numerical experiments validate effectiveness on an LQG system

Comparison feedback improves optimization in preference-based settings

Abstract

In interactive systems, feedback is often provided in the form of preference between queried options rather than precise scores, which motivates optimization methods to learn from such comparisons. In this work, we propose a preference-based optimization algorithm that relies on noisy two-point comparisons. At each iteration, the algorithm employs a uniform-sphere perturbation to generate a perturbed action and queries the resulting loss comparison to estimate a descent direction. We demonstrate that, under standard smoothness and bounded variance assumptions, the algorithm converges to a stationary point when the smoothing and step size parameters are properly chosen. Numerical experiments on an LQG system demonstrate the effectiveness of the preference-based optimization algorithm with comparison feedback.