Adaptive Future Frame Prediction with Ensemble Network

TL;DR

This paper introduces an adaptive ensemble network framework for future video frame prediction that updates online during testing, improving accuracy especially in dynamic scenes.

Contribution

It proposes a novel adaptive update framework with a pre-trained, continuously-updating, and weight estimation network for improved future frame prediction.

Findings

Outperforms existing methods in dynamic scenes

Pre-trained model achieves comparable performance to state-of-the-art

Adaptive updating enhances prediction accuracy during testing

Abstract

Future frame prediction in videos is a challenging problem because videos include complicated movements and large appearance changes. Learning-based future frame prediction approaches have been proposed in kinds of literature. A common limitation of the existing learning-based approaches is a mismatch of training data and test data. In the future frame prediction task, we can obtain the ground truth data by just waiting for a few frames. It means we can update the prediction model online in the test phase. Then, we propose an adaptive update framework for the future frame prediction task. The proposed adaptive updating framework consists of a pre-trained prediction network, a continuous-updating prediction network, and a weight estimation network. We also show that our pre-trained prediction model achieves comparable performance to the existing state-of-the-art approaches. We demonstrate that our approach outperforms existing methods especially for dynamically changing scenes.