Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling

TL;DR

This paper introduces a decoupled transformer architecture for feedforward novel view synthesis that separates semantic and spatial representations, enhancing rendering fidelity without increasing inference latency.

Contribution

It proposes a novel decoupled design with semantic and spatial tokens, plus optional supervision and modulation, improving NVS performance over prior mixed-representation models.

Findings

Achieves consistent improvements across models.

Maintains zero additional inference latency.

Effective in both decoder-only and encoder-decoder architectures.

Abstract

Transformer-based models have advanced feedforward novel view synthesis (NVS). Current architectures such as GS-LRM and LVSM mix semantic information (e.g., RGB) and spatial information (e.g., Pl\"ucker rays) into a shared feature space. Since Pl\"ucker rays naturally carry lattice-like spatial structure, these designs can make the spatial bias interfere with appearance representation and degrade rendering fidelity. To this end, we propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens. The decoupled design keeps semantic and spatial information explicit in their branches while preserving cross-branch interaction through shared attention routing. Built on this design, we introduce optional categorized supervision and bidirectional modulation: the former provides branch-specific training signals, while the latter improves interaction between the two branches. Notably, the base decoupled design introduces virtually zero additional inference latency due to its architectural design. The proposed designs achieve consistent improvements, demonstrating effectiveness across decoder-only and encoder-decoder feedforward NVS models.