Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks

TL;DR

This paper introduces Queried-Convolutions (Qonvolutions), a novel neural network modification that enhances Gaussian Splatting for high-fidelity novel view synthesis, outperforming existing radiance models like Zip-NeRF.

Contribution

It proposes Qonvolutions, leveraging neighborhood properties to improve reconstruction fidelity in Gaussian Splatting, achieving state-of-the-art results in real-world scene synthesis.

Findings

Qonvolutions improve image fidelity beyond Zip-NeRF.

QNNs enhance performance in regression and super-resolution tasks.

Gaussian Splatting with Qonvolutions achieves real-time rendering.

Abstract

Gaussian Splatting has revolutionized the field of Novel View Synthesis (NVS) with faster training and real-time rendering. However, its reconstruction fidelity still trails behind the powerful radiance models such as Zip-NeRF. Motivated by our theoretical result that both queries (such as coordinates) and neighborhood are important to learn high-fidelity signals, this paper proposes Queried-Convolutions (Qonvolutions), a simple yet powerful modification using the neighborhood properties of convolution. Qonvolutions convolve a low-fidelity signal with queries to output residual and achieve high-fidelity reconstruction. We empirically demonstrate that combining Gaussian splatting with Qonvolution neural networks (QNNs) results in state-of-the-art NVS on real-world scenes, even outperforming Zip-NeRF on image fidelity. QNNs also enhance performance of 1D regression, 2D regression and 2D super-resolution tasks.