Structure-Aware Sparse-View X-ray 3D Reconstruction

TL;DR

This paper introduces SAX-NeRF, a novel framework for sparse-view X-ray 3D reconstruction that leverages the structural nature of X-ray imaging using a transformer backbone and a specialized sampling strategy, outperforming previous methods.

Contribution

The paper presents SAX-NeRF, incorporating a Line Segment-based Transformer and a Masked Local-Global sampling strategy, along with a new large-scale dataset for improved X-ray 3D reconstruction.

Findings

Outperforms previous NeRF-based methods by 12.56 dB in novel view synthesis

Achieves 2.49 dB improvement in CT reconstruction quality

Introduces a new dataset X3D for diverse X-ray applications

Abstract

X-ray, known for its ability to reveal internal structures of objects, is expected to provide richer information for 3D reconstruction than visible light. Yet, existing neural radiance fields (NeRF) algorithms overlook this important nature of X-ray, leading to their limitations in capturing structural contents of imaged objects. In this paper, we propose a framework, Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer (Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal structures of objects in 3D space by modeling the dependencies within each line segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray sampling strategy to extract contextual and geometric information in 2D projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray applications. Experiments on X3D show that SAX-NeRF surpasses previous NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT reconstruction. Code, models, and data are released at https://github.com/caiyuanhao1998/SAX-NeRF