StROL: Stabilized and Robust Online Learning from Humans

TL;DR

StROL introduces a stability-focused online learning algorithm that adapts gradient descent rules to reliably infer human preferences in noisy and suboptimal interaction scenarios, improving accuracy and reducing regret.

Contribution

The paper presents a novel Lyapunov stability analysis-based method to modify gradient descent learning rules, enhancing robustness and convergence in online human reward inference.

Findings

StROL achieves more accurate reward inference in simulations.

StROL reduces regret compared to existing methods.

The approach maintains stability even with noisy and biased human inputs.

Abstract

Robots often need to learn the human's reward function online, during the current interaction. This real-time learning requires fast but approximate learning rules: when the human's behavior is noisy or suboptimal, current approximations can result in unstable robot learning. Accordingly, in this paper we seek to enhance the robustness and convergence properties of gradient descent learning rules when inferring the human's reward parameters. We model the robot's learning algorithm as a dynamical system over the human preference parameters, where the human's true (but unknown) preferences are the equilibrium point. This enables us to perform Lyapunov stability analysis to derive the conditions under which the robot's learning dynamics converge. Our proposed algorithm (StROL) uses these conditions to learn robust-by-design learning rules: given the original learning dynamics, StROL outputs a modified learning rule that now converges to the human's true parameters under a larger set of human inputs. In practice, these autonomously generated learning rules can correctly infer what the human is trying to convey, even when the human is noisy, biased, and suboptimal. Across simulations and a user study we find that StROL results in a more accurate estimate and less regret than state-of-the-art approaches for online reward learning. See videos and code here: https://github.com/VT-Collab/StROL_RAL