Inverse Constraint Learning and Generalization by Transferable Reward Decomposition

TL;DR

This paper introduces a transferable constraint learning algorithm that decomposes rewards to improve inverse constraint learning, enabling better generalization and higher success rates in new scenarios, validated through simulations and real-world robotics.

Contribution

The paper proposes a novel TCL algorithm that jointly infers task rewards and constraints, addressing ICL's ill-posedness and enhancing transferability and robustness.

Findings

TCL outperforms five baselines in simulated environments with up to 72% higher success rates.

TCL achieves accurate reward and constraint decomposition.

TCL demonstrates robustness on real-world robotic tasks.

Abstract

We present the problem of inverse constraint learning (ICL), which recovers constraints from demonstrations to autonomously reproduce constrained skills in new scenarios. However, ICL suffers from an ill-posed nature, leading to inaccurate inference of constraints from demonstrations. To figure it out, we introduce a transferable constraint learning (TCL) algorithm that jointly infers a task-oriented reward and a task-agnostic constraint, enabling the generalization of learned skills. Our method TCL additively decomposes the overall reward into a task reward and its residual as soft constraints, maximizing policy divergence between task- and constraint-oriented policies to obtain a transferable constraint. Evaluating our method and five baselines in three simulated environments, we show TCL outperforms state-of-the-art IRL and ICL algorithms, achieving up to a $72\%$ higher task-success rates with accurate decomposition compared to the next best approach in novel scenarios. Further, we demonstrate the robustness of TCL on two real-world robotic tasks.