Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPs

TL;DR

This paper introduces a Dynamical Variational Auto-Encoder that leverages future information to better learn causal dynamics in factored-POMDPs, improving state representations and causal graph discovery from offline data.

Contribution

The paper presents a novel DVAE framework with an extended hindsight approach that incorporates future observations to enhance causal dynamics learning in factored-POMDPs.

Findings

Outperforms history-based models in uncovering causal graphs.

Effectively captures causal Markovian dynamics from offline trajectories.

Improves state representation quality in partially observable environments.

Abstract

Learning representations of underlying environmental dynamics from partial observations is a critical challenge in machine learning. In the context of Partially Observable Markov Decision Processes (POMDPs), state representations are often inferred from the history of past observations and actions. We demonstrate that incorporating future information is essential to accurately capture causal dynamics and enhance state representations. To address this, we introduce a Dynamical Variational Auto-Encoder (DVAE) designed to learn causal Markovian dynamics from offline trajectories in a POMDP. Our method employs an extended hindsight framework that integrates past, current, and multi-step future information within a factored-POMDP setting. Empirical results reveal that this approach uncovers the causal graph governing hidden state transitions more effectively than history-based and typical hindsight-based models.