# Cubic LSTMs for Video Prediction

## TL;DR

This paper introduces CubicLSTM, a novel neural network unit with three branches designed for improved video prediction by capturing object motion and dynamics, outperforming previous methods on various datasets.

## Contribution

The paper proposes CubicLSTM, a new three-branch LSTM unit that enhances video prediction by separately modeling spatial objects and temporal motions, forming a cubic RNN architecture.

## Key findings

- CubicRNN outperforms prior methods on synthetic datasets.
- CubicRNN achieves more accurate predictions on real-world videos.
- The three-branch structure effectively captures spatial and temporal information.

## Abstract

Predicting future frames in videos has become a promising direction of research for both computer vision and robot learning communities. The core of this problem involves moving object capture and future motion prediction. While object capture specifies which objects are moving in videos, motion prediction describes their future dynamics. Motivated by this analysis, we propose a Cubic Long Short-Term Memory (CubicLSTM) unit for video prediction. CubicLSTM consists of three branches, i.e., a spatial branch for capturing moving objects, a temporal branch for processing motions, and an output branch for combining the first two branches to generate predicted frames. Stacking multiple CubicLSTM units along the spatial branch and output branch, and then evolving along the temporal branch can form a cubic recurrent neural network (CubicRNN). Experiment shows that CubicRNN produces more accurate video predictions than prior methods on both synthetic and real-world datasets.