Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling

TL;DR

This paper introduces a new framework for learning from imperfect human feedback in dueling bandits, providing theoretical regret bounds and a robust algorithm that accounts for decaying corruption in human signals.

Contribution

It develops a novel analysis framework for corruption-robust dueling bandit algorithms and proposes RoSMID, achieving near-optimal regret under decaying corruption.

Findings

Regret lower bound of Ω(max{√T, T^ρ}) established.

RoSMID algorithm attains nearly optimal regret of ˜O(max{√T, T^ρ}).

Framework applicable to other gradient-based dueling bandit algorithms.

Abstract

This paper studies Learning from Imperfect Human Feedback (LIHF), addressing the potential irrationality or imperfect perception when learning from comparative human feedback. Building on evidences that human's imperfection decays over time (i.e., humans learn to improve), we cast this problem as a concave-utility continuous-action dueling bandit but under a restricted form of corruption: i.e., the corruption scale is decaying over time as $t^{\rho-1}$ for some "imperfection rate" $\rho \in [0, 1]$. With $T$ as the total number of iterations, we establish a regret lower bound of $ \Omega(\max\{\sqrt{T}, T^{\rho}\}) $ for LIHF, even when $\rho$ is known. For the same setting, we develop the Robustified Stochastic Mirror Descent for Imperfect Dueling (RoSMID) algorithm, which achieves nearly optimal regret $\tilde{\mathcal{O}}(\max\{\sqrt{T}, T^{\rho}\})$. Core to our analysis is a novel framework for analyzing gradient-based algorithms for dueling bandit under corruption, and we demonstrate its general applicability by showing how this framework can be easily applied to obtain corruption-robust guarantees for other popular gradient-based dueling bandit algorithms. Our theoretical results are validated by extensive experiments.