Lightweight Stochastic Video Prediction via Hybrid Warping

TL;DR

This paper introduces SVPHW, a lightweight stochastic video prediction model that uses hybrid warping to improve accuracy in dynamic regions, suitable for real-time applications like autonomous driving.

Contribution

The paper presents a novel hybrid warping strategy combined with a MobileNet-based architecture for efficient, stochastic, long-term video prediction focusing on dynamic regions.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively models motion uncertainties and occlusions.

Operates in real-time with a lightweight design.

Abstract

Accurate video prediction by deep neural networks, especially for dynamic regions, is a challenging task in computer vision for critical applications such as autonomous driving, remote working, and telemedicine. Due to inherent uncertainties, existing prediction models often struggle with the complexity of motion dynamics and occlusions. In this paper, we propose a novel stochastic long-term video prediction model that focuses on dynamic regions by employing a hybrid warping strategy. By integrating frames generated through forward and backward warpings, our approach effectively compensates for the weaknesses of each technique, improving the prediction accuracy and realism of moving regions in videos while also addressing uncertainty by making stochastic predictions that account for various motions. Furthermore, considering real-time predictions, we introduce a MobileNet-based lightweight architecture into our model. Our model, called SVPHW, achieves state-of-the-art performance on two benchmark datasets.