Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices

TL;DR

This paper introduces a novel 3D Gaussian Splatting method that enables real-time, photorealistic rendering of complex 3D anatomy on lightweight devices, enhancing medical education and interactive visualization.

Contribution

The paper presents a new pipeline using compressed 3D Gaussian Splatting for efficient, high-quality view synthesis of 3D anatomy on consumer devices, overcoming current hardware limitations.

Findings

Achieves <70 MB data size for multi-GB datasets

Enables real-time, photorealistic rendering on mobile devices

Recovers structures up to voxel resolution with high fidelity

Abstract

Interactive photorealistic rendering of 3D anatomy is used in medical education to explain the structure of the human body. It is currently restricted to frontal teaching scenarios, where even with a powerful GPU and high-speed access to a large storage device where the data set is hosted, interactive demonstrations can hardly be achieved. We present the use of novel view synthesis via compressed 3D Gaussian Splatting (3DGS) to overcome this restriction, and to even enable students to perform cinematic anatomy on lightweight and mobile devices. Our proposed pipeline first finds a set of camera poses that captures all potentially seen structures in the data. High-quality images are then generated with path tracing and converted into a compact 3DGS representation, consuming < 70 MB even for data sets of multiple GBs. This allows for real-time photorealistic novel view synthesis that recovers structures up to the voxel resolution and is almost indistinguishable from the path-traced images