Tight Performance Guarantees of Imitator Policies with Continuous Actions

TL;DR

This paper extends theoretical guarantees for Behavioral Cloning to continuous actions by deriving bounds based on Wasserstein and Holder continuity, and analyzes noise injection to improve performance guarantees.

Contribution

It introduces new performance bounds for BC with continuous actions using Wasserstein and Holder continuity, and analyzes noise injection effects.

Findings

Performance gap bound based on Wasserstein distance for continuous actions.

Value function is Holder continuous under relaxed conditions.

Noise injection improves performance guarantees at the cost of bias.

Abstract

Behavioral Cloning (BC) aims at learning a policy that mimics the behavior demonstrated by an expert. The current theoretical understanding of BC is limited to the case of finite actions. In this paper, we study BC with the goal of providing theoretical guarantees on the performance of the imitator policy in the case of continuous actions. We start by deriving a novel bound on the performance gap based on Wasserstein distance, applicable for continuous-action experts, holding under the assumption that the value function is Lipschitz continuous. Since this latter condition is hardy fulfilled in practice, even for Lipschitz Markov Decision Processes and policies, we propose a relaxed setting, proving that value function is always Holder continuous. This result is of independent interest and allows obtaining in BC a general bound for the performance of the imitator policy. Finally, we analyze noise injection, a common practice in which the expert action is executed in the environment after the application of a noise kernel. We show that this practice allows deriving stronger performance guarantees, at the price of a bias due to the noise addition.