A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges

TL;DR

This survey comprehensively reviews recent advances, challenges, and applications of Inverse Constrained Reinforcement Learning, highlighting the problem definitions, algorithmic frameworks, and future research directions in the field.

Contribution

It provides a structured overview of ICRL, including formal definitions, algorithmic frameworks, and challenges across various environments and applications, serving as a key reference for researchers.

Findings

Summarizes key methods for constraint inference in ICRL.

Identifies challenges in stochastic and limited demonstration scenarios.

Discusses applications like autonomous driving and robot control.

Abstract

Inverse Constrained Reinforcement Learning (ICRL) is the task of inferring the implicit constraints that expert agents adhere to, based on their demonstration data. As an emerging research topic, ICRL has received considerable attention in recent years. This article presents a categorical survey of the latest advances in ICRL. It serves as a comprehensive reference for machine learning researchers and practitioners, as well as starters seeking to comprehend the definitions, advancements, and important challenges in ICRL. We begin by formally defining the problem and outlining the algorithmic framework that facilitates constraint inference across various scenarios. These include deterministic or stochastic environments, environments with limited demonstrations, and multiple agents. For each context, we illustrate the critical challenges and introduce a series of fundamental methods to tackle these issues. This survey encompasses discrete, virtual, and realistic environments for evaluating ICRL agents. We also delve into the most pertinent applications of ICRL, such as autonomous driving, robot control, and sports analytics. To stimulate continuing research, we conclude the survey with a discussion of key unresolved questions in ICRL that can effectively foster a bridge between theoretical understanding and practical industrial applications. The papers referenced in this survey can be found at https://github.com/Jasonxu1225/Awesome-Constraint-Inference-in-RL.