Deep 360$^\circ$ Optical Flow Estimation Based on Multi-Projection Fusion

TL;DR

This paper introduces a novel multi-projection fusion framework for 360-degree optical flow estimation using deep neural networks, addressing distortions in panoramic images and outperforming existing methods.

Contribution

It proposes a new multi-projection fusion approach and provides the first large-scale panoramic optical flow dataset for training and evaluation.

Findings

Our method outperforms existing 360-degree optical flow techniques.

The multi-projection fusion effectively combines complementary information from different projections.

The new dataset enables better training and benchmarking of panoramic optical flow models.

Abstract

Optical flow computation is essential in the early stages of the video processing pipeline. This paper focuses on a less explored problem in this area, the 360$^\circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. To address the distortions of panoramic representations when applying convolutional neural networks, we propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. It learns to combine the complementary information in the optical flow results under different projections. We also build the first large-scale panoramic optical flow dataset to support the training of neural networks and the evaluation of panoramic optical flow estimation methods. The experimental results on our dataset demonstrate that our method outperforms the existing methods and other alternative deep networks that were developed for processing 360{\deg} content.