Sketch and Patch: Efficient 3D Gaussian Representation for Man-Made Scenes

TL;DR

This paper introduces a hybrid 3D Gaussian representation that categorizes Gaussians into sketch and patch types, significantly reducing storage needs while maintaining high-quality photorealistic scene rendering.

Contribution

It proposes a novel categorization and encoding scheme for 3D Gaussians, improving storage efficiency and rendering quality in 3D scene representations.

Findings

Achieves up to 32.62% improvement in PSNR

Reduces model size to 2.3% of original for indoor scenes

Enhances volumetric rendering quality across diverse scenes

Abstract

3D Gaussian Splatting (3DGS) has emerged as a promising representation for photorealistic rendering of 3D scenes. However, its high storage requirements pose significant challenges for practical applications. We observe that Gaussians exhibit distinct roles and characteristics that are analogous to traditional artistic techniques -- Like how artists first sketch outlines before filling in broader areas with color, some Gaussians capture high-frequency features like edges and contours; While other Gaussians represent broader, smoother regions, that are analogous to broader brush strokes that add volume and depth to a painting. Based on this observation, we propose a novel hybrid representation that categorizes Gaussians into (i) Sketch Gaussians, which define scene boundaries, and (ii) Patch Gaussians, which cover smooth regions. Sketch Gaussians are efficiently encoded using parametric models, leveraging their geometric coherence, while Patch Gaussians undergo optimized pruning, retraining, and vector quantization to maintain volumetric consistency and storage efficiency. Our comprehensive evaluation across diverse indoor and outdoor scenes demonstrates that this structure-aware approach achieves up to 32.62% improvement in PSNR, 19.12% in SSIM, and 45.41% in LPIPS at equivalent model sizes, and correspondingly, for an indoor scene, our model maintains the visual quality with 2.3% of the original model size.