Low Latency Point Cloud Rendering with Learned Splatting

TL;DR

This paper introduces a neural network-based framework for real-time, high-quality point cloud rendering that achieves low latency and generalizes across different scenes without per-scene optimization.

Contribution

It proposes a novel learned splatting method using neural networks to estimate Gaussian representations for point clouds, enabling fast and high-fidelity rendering.

Findings

Achieves real-time rendering with high visual quality.

Demonstrates robustness to compression artifacts.

Generalizes well across different scene contents.

Abstract

Point cloud is a critical 3D representation with many emerging applications. Because of the point sparsity and irregularity, high-quality rendering of point clouds is challenging and often requires complex computations to recover the continuous surface representation. On the other hand, to avoid visual discomfort, the motion-to-photon latency has to be very short, under 10 ms. Existing rendering solutions lack in either quality or speed. To tackle these challenges, we present a framework that unlocks interactive, free-viewing and high-fidelity point cloud rendering. We train a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and use differentiable surface splatting to render smooth texture and surface normal for arbitrary views. Our approach does not require per-scene optimization, and enable real-time rendering of dynamic point cloud. Experimental results demonstrate the proposed solution enjoys superior visual quality and speed, as well as generalizability to different scene content and robustness to compression artifacts. The code is available at https://github.com/huzi96/gaussian-pcloud-render .