Robustness in the Face of Partial Identifiability in Reward Learning

TL;DR

This paper addresses the challenge of partial reward identifiability in Reward Learning by proposing a robust framework that maximizes worst-case performance, with theoretical guarantees and a practical algorithm called Rob-ReL.

Contribution

It introduces a general framework for quantifying performance loss due to reward ambiguity and develops Rob-ReL, a robust algorithm with theoretical complexity guarantees.

Findings

Rob-ReL effectively handles reward ambiguity in preference-based reward learning.

Theoretical guarantees on sample and iteration complexity for Rob-ReL.

Proof-of-concept experiments demonstrate robustness in partial identifiability scenarios.

Abstract

In Reward Learning (ReL), we are given feedback on an unknown target reward, and the goal is to use this information to recover it in order to carry out some downstream application, e.g., planning. When the feedback is not informative enough, the target reward is only partially identifiable, i.e., there exists a set of rewards, called the feasible set, that are equally plausible candidates for the target reward. In these cases, the ReL algorithm might recover a reward function different from the target reward, possibly leading to a failure in the application. In this paper, we introduce a general ReL framework that permits to quantify the drop in "performance" suffered in the considered application because of identifiability issues. Building on this, we propose a robust approach to address the identifiability problem in a principled way, by maximizing the "performance" with respect to the worst-case reward in the feasible set. We then develop Rob-ReL, a ReL algorithm that applies this robust approach to the subset of ReL problems aimed at assessing a preference between two policies, and we provide theoretical guarantees on sample and iteration complexity for Rob-ReL. We conclude with a proof-of-concept experiment to illustrate the considered setting.