Scaling View Synthesis Transformers

TL;DR

This paper systematically studies the scaling laws of geometry-free view synthesis transformers, introduces the SVSM architecture, and demonstrates its compute efficiency and superior performance on real-world benchmarks.

Contribution

It reveals that encoder-decoder architectures can be compute-optimal for view synthesis and provides design principles for training such models efficiently.

Findings

Encoder-decoder models scale as effectively as decoder-only models.

SVSM achieves a better performance-compute Pareto frontier.

SVSM surpasses previous state-of-the-art with less training compute.

Abstract

Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.