InsideOut: Integrated RGB-Radiative Gaussian Splatting for Comprehensive 3D Object Representation

TL;DR

InsideOut is a novel 3D Gaussian splatting method that fuses RGB surface details with subsurface X-ray structures, enabling comprehensive internal and external object visualization for medical, cultural, and industrial applications.

Contribution

It introduces a new framework that combines RGB and X-ray data through hierarchical fitting and an X-ray reference loss, extending 3D Gaussian splatting to multi-modal internal-external object representation.

Findings

Effective fusion of RGB and X-ray data for 3D modeling

Enhanced visualization of internal structures in 3D objects

Applicable to medical diagnostics, heritage, and manufacturing

Abstract

We introduce InsideOut, an extension of 3D Gaussian splatting (3DGS) that bridges the gap between high-fidelity RGB surface details and subsurface X-ray structures. The fusion of RGB and X-ray imaging is invaluable in fields such as medical diagnostics, cultural heritage restoration, and manufacturing. We collect new paired RGB and X-ray data, perform hierarchical fitting to align RGB and X-ray radiative Gaussian splats, and propose an X-ray reference loss to ensure consistent internal structures. InsideOut effectively addresses the challenges posed by disparate data representations between the two modalities and limited paired datasets. This approach significantly extends the applicability of 3DGS, enhancing visualization, simulation, and non-destructive testing capabilities across various domains.