Learning Constraint Network from Demonstrations via Positive-Unlabeled Learning with Memory Replay

TL;DR

This paper introduces a positive-unlabeled learning framework with memory replay to infer complex, nonlinear constraints from expert demonstrations, enabling safer and more accurate planning in complex environments.

Contribution

It presents a novel PU learning approach combined with iterative policy updates and memory replay to infer arbitrary constraints from demonstrations, overcoming limitations of prior methods.

Findings

Successfully infers nonlinear constraints in Mujoco environments

Outperforms baseline in constraint accuracy

Enhances policy safety and robustness

Abstract

Planning for a wide range of real-world tasks necessitates to know and write all constraints. However, instances exist where these constraints are either unknown or challenging to specify accurately. A possible solution is to infer the unknown constraints from expert demonstration. The majority of prior works limit themselves to learning simple linear constraints, or require strong knowledge of the true constraint parameterization or environmental model. To mitigate these problems, this paper presents a positive-unlabeled (PU) learning approach to infer a continuous, arbitrary and possibly nonlinear, constraint from demonstration. From a PU learning view, We treat all data in demonstrations as positive (feasible) data, and learn a (sub)-optimal policy to generate high-reward-winning but potentially infeasible trajectories, which serve as unlabeled data containing both feasible and infeasible states. Under an assumption on data distribution, a feasible-infeasible classifier (i.e., constraint model) is learned from the two datasets through a postprocessing PU learning technique. The entire method employs an iterative framework alternating between updating the policy, which generates and selects higher-reward policies, and updating the constraint model. Additionally, a memory buffer is introduced to record and reuse samples from previous iterations to prevent forgetting. The effectiveness of the proposed method is validated in two Mujoco environments, successfully inferring continuous nonlinear constraints and outperforming a baseline method in terms of constraint accuracy and policy safety.