Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priors

TL;DR

This paper introduces efficient algorithms for recovering time-varying reward functions in max entropy reinforcement learning by leveraging sparsity and rank priors, improving accuracy and computational feasibility.

Contribution

It formulates reward identification as sparsification and rank minimization problems, providing polynomial-time algorithms with practical applications.

Findings

Algorithms accurately recover rewards from demonstrations.

Methods outperform baseline approaches in reward recovery.

Reconstructed rewards generalize well to new policies.

Abstract

In this paper, we consider the problem of recovering time-varying reward functions from either optimal policies or demonstrations coming from a max entropy reinforcement learning problem. This problem is highly ill-posed without additional assumptions on the underlying rewards. However, in many applications, the rewards are indeed parsimonious, and some prior information is available. We consider two such priors on the rewards: 1) rewards are mostly constant and they change infrequently, 2) rewards can be represented by a linear combination of a small number of feature functions. We first show that the reward identification problem with the former prior can be recast as a sparsification problem subject to linear constraints. Moreover, we give a polynomial-time algorithm that solves this sparsification problem exactly. Then, we show that identifying rewards representable with the minimum number of features can be recast as a rank minimization problem subject to linear constraints, for which convex relaxations of rank can be invoked. In both cases, these observations lead to efficient optimization-based reward identification algorithms. Several examples are given to demonstrate the accuracy of the recovered rewards as well as their generalizability.