Robust Behavior Cloning Via Global Lipschitz Regularization

TL;DR

This paper introduces a global Lipschitz regularization method to improve the robustness of behavior cloning policies against observation perturbations, providing theoretical robustness guarantees and empirical validation across multiple environments.

Contribution

It proposes a novel Lipschitz regularization approach to enhance policy robustness in behavior cloning, with a method to construct Lipschitz neural networks ensuring robustness guarantees.

Findings

Lipschitz regularization improves policy robustness against bounded perturbations.

The approach provides a formal robustness certificate for the learned policies.

Empirical results demonstrate effectiveness across various Gymnasium environments.

Abstract

Behavior Cloning (BC) is an effective imitation learning technique and has even been adopted in some safety-critical domains such as autonomous vehicles. BC trains a policy to mimic the behavior of an expert by using a dataset composed of only state-action pairs demonstrated by the expert, without any additional interaction with the environment. However, During deployment, the policy observations may contain measurement errors or adversarial disturbances. Since the observations may deviate from the true states, they can mislead the agent into making sub-optimal actions. In this work, we use a global Lipschitz regularization approach to enhance the robustness of the learned policy network. We then show that the resulting global Lipschitz property provides a robustness certificate to the policy with respect to different bounded norm perturbations. Then, we propose a way to construct a Lipschitz neural network that ensures the policy robustness. We empirically validate our theory across various environments in Gymnasium. Keywords: Robust Reinforcement Learning; Behavior Cloning; Lipschitz Neural Network