IBGS: Image-Based Gaussian Splatting

TL;DR

This paper introduces Image-Based Gaussian Splatting, a novel method that enhances 3D Gaussian Splatting for view synthesis by using high-resolution images to better capture details and view-dependent effects, outperforming previous methods.

Contribution

It proposes a new approach that combines source images with Gaussian Splatting to improve detail and view-dependent effects without extra storage overhead.

Findings

Significantly improves rendering quality over prior Gaussian Splatting methods.

Effectively captures high-frequency details and view-dependent effects.

Maintains low storage requirements despite enhanced rendering capabilities.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a fast, high-quality method for novel view synthesis (NVS). However, its use of low-degree spherical harmonics limits its ability to capture spatially varying color and view-dependent effects such as specular highlights. Existing works augment Gaussians with either a global texture map, which struggles with complex scenes, or per-Gaussian texture maps, which introduces high storage overhead. We propose Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling. Specifically, we model each pixel color as a combination of a base color from standard 3DGS rendering and a learned residual inferred from neighboring training images. This promotes accurate surface alignment and enables rendering images of high-frequency details and accurate view-dependent effects. Experiments on standard NVS benchmarks show that our method significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.