Learning Accurate Extended-Horizon Predictions of High Dimensional Trajectories

TL;DR

This paper introduces a novel predictive coding architecture capable of accurate long-horizon open-loop predictions of high-dimensional trajectories, improving immediate prediction accuracy and reducing sample complexity in reinforcement learning tasks.

Contribution

The paper presents a new predictive coding model that runs open loop during training and maps initial observations to hidden states for instant accurate predictions, advancing trajectory prediction methods.

Findings

Achieves accurate long-horizon predictions of Doppler radar data during Mars landing.

Reduces sample complexity by 2X using a Dyna-style algorithm with PPO.

Outperforms standard predictive coding models in trajectory prediction tasks.

Abstract

We present a novel predictive model architecture based on the principles of predictive coding that enables open loop prediction of future observations over extended horizons. There are two key innovations. First, whereas current methods typically learn to make long-horizon open-loop predictions using a multi-step cost function, we instead run the model open loop in the forward pass during training. Second, current predictive coding models initialize the representation layer's hidden state to a constant value at the start of an episode, and consequently typically require multiple steps of interaction with the environment before the model begins to produce accurate predictions. Instead, we learn a mapping from the first observation in an episode to the hidden state, allowing the trained model to immediately produce accurate predictions. We compare the performance of our architecture to a standard predictive coding model and demonstrate the ability of the model to make accurate long horizon open-loop predictions of simulated Doppler radar altimeter readings during a six degree of freedom Mars landing. Finally, we demonstrate a 2X reduction in sample complexity by using the model to implement a Dyna style algorithm to accelerate policy learning with proximal policy optimization.