Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs

TL;DR

This paper extends inverse models to learn agent-centric state representations in high-dimensional, non-Markovian environments, providing theoretical insights and empirical evaluations of different algorithms.

Contribution

It introduces a generalized inverse model approach for state discovery in non-Markovian settings and offers both theoretical analysis and empirical validation.

Findings

Inverse models can be adapted for non-Markovian environments.

Past actions can improve or hinder state discovery depending on their use.

Empirical results demonstrate the effectiveness and limitations of proposed methods.

Abstract

Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.