Revealing Disocclusions in Temporal View Synthesis through Infilling Vector Prediction

TL;DR

This paper introduces a novel infilling vector prediction method for temporal view synthesis, leveraging depth and temporal cues to improve the prediction of disoccluded regions in future video frames.

Contribution

The paper proposes a new approach that predicts infilling vectors using a learned temporal prior and depth information, enhancing disocclusion handling in view synthesis.

Findings

Achieves superior quantitative performance over existing methods.

Demonstrates improved qualitative results in disocclusion infilling.

Validates approach on a large-scale dataset and SceneNet RGB-D.

Abstract

We consider the problem of temporal view synthesis, where the goal is to predict a future video frame from the past frames using knowledge of the depth and relative camera motion. In contrast to revealing the disoccluded regions through intensity based infilling, we study the idea of an infilling vector to infill by pointing to a non-disoccluded region in the synthesized view. To exploit the structure of disocclusions created by camera motion during their infilling, we rely on two important cues, temporal correlation of infilling directions and depth. We design a learning framework to predict the infilling vector by computing a temporal prior that reflects past infilling directions and a normalized depth map as input to the network. We conduct extensive experiments on a large scale dataset we build for evaluating temporal view synthesis in addition to the SceneNet RGB-D dataset. Our experiments demonstrate that our infilling vector prediction approach achieves superior quantitative and qualitative infilling performance compared to other approaches in literature.