Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis

TL;DR

This paper introduces a novel multiple homography image (MHI) representation and a two-stage neural network for view synthesis from stereo images with rotations, outperforming existing methods especially in rotational scenarios.

Contribution

The paper proposes the MHI representation and a two-stage network to generalize view synthesis to unstructured views with rotations, addressing depth-dependency issues.

Findings

Achieves superior qualitative view synthesis results.

Outperforms state-of-the-art methods quantitatively.

Effective in scenarios with camera rotations.

Abstract

This paper studies the problem of view synthesis with certain amount of rotations from a pair of images, what we called stereo unstructured magnification. While the multi-plane image representation is well suited for view synthesis with depth invariant, how to generalize it to unstructured views remains a significant challenge. This is primarily due to the depth-dependency caused by camera frontal parallel representation. Here we propose a novel multiple homography image (MHI) representation, comprising of a set of scene planes with fixed normals and distances. A two-stage network is developed for novel view synthesis. Stage-1 is an MHI reconstruction module that predicts the MHIs and composites layered multi-normal images along the normal direction. Stage-2 is a normal-blending module to find blending weights. We also derive an angle-based cost to guide the blending of multi-normal images by exploiting per-normal geometry. Compared with the state-of-the-art methods, our method achieves superior performance for view synthesis qualitatively and quantitatively, especially for cases when the cameras undergo rotations.