Learning Transition Models with Time-delayed Causal Relations

TL;DR

This paper presents an algorithm for discovering delayed causal relations in robotic observations, enhancing data efficiency and interpretability in model-based reinforcement learning by incorporating memory units that track past events.

Contribution

The paper introduces a novel method that identifies and models implicit, time-delayed causal relations using hidden memory variables, improving RL performance.

Findings

Significant improvement over existing RL methods in robotic tasks

Effective identification of relevant past events through information gain

Enhanced data efficiency and interpretability of models

Abstract

This paper introduces an algorithm for discovering implicit and delayed causal relations between events observed by a robot at arbitrary times, with the objective of improving data-efficiency and interpretability of model-based reinforcement learning (RL) techniques. The proposed algorithm initially predicts observations with the Markov assumption, and incrementally introduces new hidden variables to explain and reduce the stochasticity of the observations. The hidden variables are memory units that keep track of pertinent past events. Such events are systematically identified by their information gains. The learned transition and reward models are then used for planning. Experiments on simulated and real robotic tasks show that this method significantly improves over current RL techniques.