Drop-In Perceptual Optimization for 3D Gaussian Splatting

TL;DR

This paper introduces WD-R, a perceptual loss for 3D Gaussian Splatting that significantly improves visual quality and texture recovery, validated through extensive human studies and outperforming existing methods.

Contribution

We systematically evaluate perceptual optimization strategies for 3DGS and propose WD-R, a novel regularized Wasserstein Distortion loss that enhances perceptual quality and texture detail.

Findings

WD-R outperforms original 3DGS loss by 2.3x in human preference.

WD-R achieves state-of-the-art LPIPS, DISTS, and FID scores.

Replacing the loss with WD-R improves perceptual quality across frameworks and reduces scene bitrate by 50%.

Abstract

Despite their output being ultimately consumed by human viewers, 3D Gaussian Splatting (3DGS) methods often rely on ad-hoc combinations of pixel-level losses, resulting in blurry renderings. To address this, we systematically explore perceptual optimization strategies for 3DGS by searching over a diverse set of distortion losses. We conduct the first-of-its-kind large-scale human subjective study on 3DGS, involving 39,320 pairwise ratings across several datasets and 3DGS frameworks. A regularized version of Wasserstein Distortion, which we call WD-R, emerges as the clear winner, excelling at recovering fine textures without incurring a higher splat count. WD-R is preferred by raters more than $2.3\times$ over the original 3DGS loss, and $1.5\times$ over current best method Perceptual-GS. WD-R also consistently achieves state-of-the-art LPIPS, DISTS, and FID scores across various datasets, and generalizes across recent frameworks, such as Mip-Splatting and Scaffold-GS, where replacing the original loss with WD-R consistently enhances perceptual quality within a similar resource budget (number of splats for Mip-Splatting, model size for Scaffold-GS), and leads to reconstructions being preferred by human raters $1.8\times$ and $3.6\times$, respectively. We also find that this carries over to the task of 3DGS scene compression, with $\approx 50\%$ bitrate savings for comparable perceptual metric performance.