Frequency-Adaptive Sharpness Regularization for Improving 3D Gaussian Splatting Generalization

TL;DR

This paper introduces Frequency-Adaptive Sharpness Regularization (FASR), a novel training method for 3D Gaussian Splatting that enhances its ability to generalize to unseen viewpoints by adaptively controlling regularization based on local image frequency.

Contribution

The paper proposes FASR, a new regularization approach that improves 3DGS generalization by adaptively balancing sharpness reduction according to local image frequency, addressing limitations of existing methods.

Findings

FASR improves generalization across multiple datasets.

It prevents artifacts and preserves fine details better than SAM.

FASR consistently outperforms baseline methods in experiments.

Abstract

Despite 3D Gaussian Splatting (3DGS) excelling in most configurations, it lacks generalization across novel viewpoints in a few-shot scenario because it overfits to the sparse observations. We revisit 3DGS optimization from a machine learning perspective, framing novel view synthesis as a generalization problem to unseen viewpoints-an underexplored direction. We propose Frequency-Adaptive Sharpness Regularization (FASR), which reformulates the 3DGS training objective, thereby guiding 3DGS to converge toward a better generalization solution. Although Sharpness-Aware Minimization (SAM) similarly reduces the sharpness of the loss landscape to improve generalization of classification models, directly employing it to 3DGS is suboptimal due to the discrepancy between the tasks. Specifically, it hinders reconstructing high-frequency details due to excessive regularization, while reducing its strength leads to under-penalizing sharpness. To address this, we reflect the local frequency of images to set the regularization weight and the neighborhood radius when estimating the local sharpness. It prevents floater artifacts in novel viewpoints and reconstructs fine details that SAM tends to oversmooth. Across datasets with various configurations, our method consistently improves a wide range of baselines. Code will be available at https://bbangsik13.github.io/FASR.