Initial State Interventions for Deconfounded Imitation Learning

TL;DR

This paper introduces a novel initial state intervention method to improve imitation learning by reducing causal confusion without requiring expert queries or causal graphs, demonstrated on control systems.

Contribution

It proposes a new masking algorithm leveraging initial state interventions to mitigate causal confusion in imitation learning, with theoretical guarantees and practical validation.

Findings

The algorithm reduces causal confusion in imitation learning.

Theoretical proof of conservative masking behavior.

Successful application to CartPole and Reacher environments.

Abstract

Imitation learning suffers from causal confusion. This phenomenon occurs when learned policies attend to features that do not causally influence the expert actions but are instead spuriously correlated. Causally confused agents produce low open-loop supervised loss but poor closed-loop performance upon deployment. We consider the problem of masking observed confounders in a disentangled representation of the observation space. Our novel masking algorithm leverages the usual ability to intervene in the initial system state, avoiding any requirement involving expert querying, expert reward functions, or causal graph specification. Under certain assumptions, we theoretically prove that this algorithm is conservative in the sense that it does not incorrectly mask observations that causally influence the expert; furthermore, intervening on the initial state serves to strictly reduce excess conservatism. The masking algorithm is applied to behavior cloning for two illustrative control systems: CartPole and Reacher.