LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference

TL;DR

LoLep is a novel single-view view synthesis method that regresses locally-learned planes and uses self-attention for occlusion inference, achieving state-of-the-art results without depth data.

Contribution

The paper introduces a new approach combining locally-learned planes, disparity sampling, and self-attention to improve scene representation and view synthesis from a single image.

Findings

Achieves state-of-the-art view synthesis results on multiple datasets.

Reduces LPIPS by 4.8%-9.0% compared to MINE.

Significantly improves real-world image synthesis performance.

Abstract

We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately, thus generating better novel views. Without the depth information, regressing appropriate plane locations is a challenging problem. To solve this issue, we pre-partition the disparity space into bins and design a disparity sampler to regress local offsets for multiple planes in each bin. However, only using such a sampler makes the network not convergent; we further propose two optimizing strategies that combine with different disparity distributions of datasets and propose an occlusion-aware reprojection loss as a simple yet effective geometric supervision technique. We also introduce a self-attention mechanism to improve occlusion inference and present a Block-Sampling Self-Attention (BS-SA) module to address the problem of applying self-attention to large feature maps. We demonstrate the effectiveness of our approach and generate state-of-the-art results on different datasets. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%. We also evaluate the performance on real-world images and demonstrate the benefits.