Neural Radiance Projection

TL;DR

Neural Radiance Projection (NeRP) leverages GANs to synthesize realistic X-ray images from 3D data, improving segmentation accuracy and addressing data scarcity, ambiguity, and class imbalance issues in X-ray image segmentation.

Contribution

The paper introduces NeRP, a novel method that uses GANs to generate physics-based X-ray images from 3D data, enhancing segmentation performance and data availability.

Findings

VRRs are more photo-realistic than other projection methods.

NeRP outputs outperform vanilla UNet models on the same data.

The approach effectively addresses data scarcity and class imbalance.

Abstract

The proposed method, Neural Radiance Projection (NeRP), addresses the three most fundamental shortages of training such a convolutional neural network on X-ray image segmentation: dealing with missing/limited human-annotated datasets; ambiguity on the per-pixel label; and the imbalance across positive- and negative- classes distribution. By harnessing a generative adversarial network, we can synthesize a massive amount of physics-based X-ray images, so-called Variationally Reconstructed Radiographs (VRRs), alongside their segmentation from more accurate labeled 3D Computed Tomography data. As a result, VRRs present more faithfully than other projection methods in terms of photo-realistic metrics. Adding outputs from NeRP also surpasses the vanilla UNet models trained on the same pairs of X-ray images.