Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention

TL;DR

Efficient-LVSM introduces a dual-stream transformer architecture for large view synthesis that reduces computational complexity, improves speed, and enhances performance over previous models like LVSM.

Contribution

It proposes a decoupled co-refinement attention mechanism that improves efficiency and generalization in large view synthesis models.

Findings

Achieves 29.86 dB PSNR on RealEstate10K with 2 views, surpassing LVSM.

Offers 2x faster training convergence and 4.4x faster inference.

State-of-the-art performance on multiple benchmarks with strong zero-shot generalization.

Abstract

Feedforward models for novel view synthesis (NVS) have recently advanced by transformer-based methods like LVSM, using attention among all input and target views. In this work, we argue that its full self-attention design is suboptimal, suffering from quadratic complexity with respect to the number of input views and rigid parameter sharing among heterogeneous tokens. We propose Efficient-LVSM, a dual-stream architecture that avoids these issues with a decoupled co-refinement mechanism. It applies intra-view self-attention for input views and self-then-cross attention for target views, eliminating unnecessary computation. Efficient-LVSM achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed. Efficient-LVSM achieves state-of-the-art performance on multiple benchmarks, exhibits strong zero-shot generalization to unseen view counts, and enables incremental inference with KV-cache, thanks to its decoupled designs.