Apprenticeship learning with prior beliefs using inverse optimization

TL;DR

This paper explores the connection between inverse optimization and inverse reinforcement learning for Markov decision processes, introducing a regularized framework that incorporates prior beliefs and improves the learning of cost functions and policies.

Contribution

It presents a novel regularized inverse optimization framework for IRL and apprenticeship learning, addressing ill-posedness and demonstrating convergence with stochastic mirror descent.

Findings

Regularization improves cost vector and policy learning.

The framework generalizes existing apprenticeship learning models.

Numerical experiments validate the importance of regularization.

Abstract

The relationship between inverse reinforcement learning (IRL) and inverse optimization (IO) for Markov decision processes (MDPs) has been relatively underexplored in the literature, despite addressing the same problem. In this work, we revisit the relationship between the IO framework for MDPs, IRL, and apprenticeship learning (AL). We incorporate prior beliefs on the structure of the cost function into the IRL and AL problems, and demonstrate that the convex-analytic view of the AL formalism emerges as a relaxation of our framework. Notably, the AL formalism is a special case in our framework when the regularization term is absent. Focusing on the suboptimal expert setting, we formulate the AL problem as a regularized min-max problem. The regularizer plays a key role in addressing the ill-posedness of IRL by guiding the search for plausible cost functions. To solve the resulting regularized-convex-concave-min-max problem, we use stochastic mirror descent (SMD) and establish convergence bounds for the proposed method. Numerical experiments highlight the critical role of regularization in learning cost vectors and apprentice policies.