A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

TL;DR

This survey comprehensively reviews inverse reinforcement learning, highlighting its challenges, current methods, extensions, and open research questions to guide future research and practical applications.

Contribution

It provides a structured overview of IRL challenges, methods, extensions, and recent progress, serving as a valuable reference for researchers and practitioners.

Findings

Identifies key challenges like inference accuracy and scalability.

Discusses methods to handle perception inaccuracies and multiple rewards.

Highlights open questions and future directions in IRL research.

Abstract

Inverse reinforcement learning (IRL) is the problem of inferring the reward function of an agent, given its policy or observed behavior. Analogous to RL, IRL is perceived both as a problem and as a class of methods. By categorically surveying the current literature in IRL, this article serves as a reference for researchers and practitioners of machine learning and beyond to understand the challenges of IRL and select the approaches best suited for the problem on hand. The survey formally introduces the IRL problem along with its central challenges such as the difficulty in performing accurate inference and its generalizability, its sensitivity to prior knowledge, and the disproportionate growth in solution complexity with problem size. The article elaborates how the current methods mitigate these challenges. We further discuss the extensions to traditional IRL methods for handling: inaccurate and incomplete perception, an incomplete model, multiple reward functions, and nonlinear reward functions. This survey concludes the discussion with some broad advances in the research area and currently open research questions.