Learning Causal State Representations of Partially Observable Environments

TL;DR

This paper introduces an algorithm to learn causal state representations in POMDPs using RNNs, enabling more efficient policy learning in environments with complex observations, backed by theoretical guarantees and empirical validation.

Contribution

It proposes a novel method to approximate causal states with RNNs, connecting causal inference with reinforcement learning, and provides theoretical bounds on optimality.

Findings

Learned causal states improve policy learning efficiency.

Theoretical guarantees relate causal states to bisimulation.

Empirical results show superior performance over prior methods.

Abstract

Intelligent agents can cope with sensory-rich environments by learning task-agnostic state abstractions. In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP). Our method learns approximate causal state representations from RNNs trained to predict subsequent observations given the history. We demonstrate that these learned state representations are useful for learning policies efficiently in reinforcement learning problems with rich observation spaces. We connect causal states with causal feature sets from the causal inference literature, and also provide theoretical guarantees on the optimality of the continuous version of this causal state representation under Lipschitz assumptions by proving equivalence to bisimulation, a relation between behaviorally equivalent systems. This allows for lower bounds on the optimal value function of the learned representation, which is tight given certain assumptions. Finally, we empirically evaluate causal state representations using multiple partially observable tasks and compare with prior methods.