Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement

TL;DR

Sphinx is a hybrid inference framework for novel view synthesis that combines regression-based initialization with selective refinement, achieving diffusion-level quality with significantly reduced computation and flexible performance-quality trade-offs.

Contribution

Sphinx introduces a training-free, hybrid approach that guides diffusion models with regression-based initialization and adaptive refinement, enabling efficient, high-quality novel view synthesis.

Findings

Achieves 1.8x speedup over diffusion models

Maintains less than 5% perceptual degradation

Establishes a new Pareto frontier between quality and latency

Abstract

Novel View Synthesis (NVS) is the task of generating new images of a scene from viewpoints that were not part of the original input. Diffusion-based NVS can generate high-quality, temporally consistent images, however, remains computationally prohibitive. Conversely, regression-based NVS offers suboptimal generation quality despite requiring significantly lower compute; leaving the design objective of a high-quality, inference-efficient NVS framework an open challenge. To close this critical gap, we present Sphinx, a training-free hybrid inference framework that achieves diffusion-level fidelity at a significantly lower compute. Sphinx proposes to use regression-based fast initialization to guide and reduce the denoising workload for the diffusion model. Additionally, it integrates selective refinement with adaptive noise scheduling, allowing more compute to uncertain regions and frames. This enables Sphinx to provide flexible navigation of the performance-quality trade-off, allowing adaptation to latency and fidelity requirements for dynamically changing inference scenarios. Our evaluation shows that Sphinx achieves an average 1.8x speedup over diffusion model inference with negligible perceptual degradation of less than 5%, establishing a new Pareto frontier between quality and latency in NVS serving.