Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering

TL;DR

This paper introduces a novel end-to-end learned point cloud compression method that directly produces a scalable bit stream for real-time rendering of high-quality 3D scenes, enhancing AR/VR streaming.

Contribution

It proposes a joint optimization scheme for encoding and decoding point clouds into directly renderable 3D Gaussians with scalable quality levels, improving efficiency and rendering quality.

Findings

Significantly improves rendering quality over existing methods.

Reduces decoding and rendering time for high-quality point clouds.

Supports real-time color decoding and scalable detail levels.

Abstract

Point cloud is a promising 3D representation for volumetric streaming in emerging AR/VR applications. Despite recent advances in point cloud compression, decoding and rendering high-quality images from lossy compressed point clouds is still challenging in terms of quality and complexity, making it a major roadblock to achieve real-time 6-Degree-of-Freedom video streaming. In this paper, we address this problem by developing a point cloud compression scheme that generates a bit stream that can be directly decoded to renderable 3D Gaussians. The encoder and decoder are jointly optimized to consider both bit-rates and rendering quality. It significantly improves the rendering quality while substantially reducing decoding and rendering time, compared to existing point cloud compression methods. Furthermore, the proposed scheme generates a scalable bit stream, allowing multiple levels of details at different bit-rate ranges. Our method supports real-time color decoding and rendering of high quality point clouds, thus paving the way for interactive 3D streaming applications with free view points.