Towards Generalized Inverse Reinforcement Learning

TL;DR

This paper introduces a generalized inverse reinforcement learning framework that learns MDP components from observed, possibly suboptimal, policies with unknown or partially known elements, using a new formulation and heuristic algorithm.

Contribution

It formulates the GIRL problem considering uncertain MDP components and proposes a fast heuristic algorithm to solve it, addressing key challenges in the field.

Findings

The proposed formulation effectively captures the GIRL problem.

The heuristic algorithm demonstrates good performance on finite and infinite state problems.

Numerical results validate the approach's merit.

Abstract

This paper studies generalized inverse reinforcement learning (GIRL) in Markov decision processes (MDPs), that is, the problem of learning the basic components of an MDP given observed behavior (policy) that might not be optimal. These components include not only the reward function and transition probability matrices, but also the action space and state space that are not exactly known but are known to belong to given uncertainty sets. We address two key challenges in GIRL: first, the need to quantify the discrepancy between the observed policy and the underlying optimal policy; second, the difficulty of mathematically characterizing the underlying optimal policy when the basic components of an MDP are unobservable or partially observable. Then, we propose the mathematical formulation for GIRL and develop a fast heuristic algorithm. Numerical results on both finite and infinite state problems show the merit of our formulation and algorithm.