RadRotator: 3D Rotation of Radiographs with Diffusion Models

TL;DR

RadRotator introduces a diffusion model-based method for rotating 2D radiographs in 3D space, enabling comprehensive visualization from any viewpoint, and addresses previous limitations by using classifier-free guidance and pixel intensity transformations.

Contribution

The paper presents a novel diffusion model approach for 3D rotation of radiographs, improving over GANs and style transfer methods for medical image visualization.

Findings

Diffusion models outperform GANs in image quality and mode coverage.

Pixel intensity transformation enhances model robustness to input variations.

The method enables reliable 3D visualization from 2D radiographs.

Abstract

Transforming two-dimensional (2D) images into three-dimensional (3D) volumes is a well-known yet challenging problem for the computer vision community. In the medical domain, a few previous studies attempted to convert two or more input radiographs into computed tomography (CT) volumes. Following their effort, we introduce a diffusion model-based technology that can rotate the anatomical content of any input radiograph in 3D space, potentially enabling the visualization of the entire anatomical content of the radiograph from any viewpoint in 3D. Similar to previous studies, we used CT volumes to create Digitally Reconstructed Radiographs (DRRs) as the training data for our model. However, we addressed two significant limitations encountered in previous studies: 1. We utilized conditional diffusion models with classifier-free guidance instead of Generative Adversarial Networks (GANs) to achieve higher mode coverage and improved output image quality, with the only trade-off being slower inference time, which is often less critical in medical applications; and 2. We demonstrated that the unreliable output of style transfer deep learning (DL) models, such as Cycle-GAN, to transfer the style of actual radiographs to DRRs could be replaced with a simple yet effective training transformation that randomly changes the pixel intensity histograms of the input and ground-truth imaging data during training. This transformation makes the diffusion model agnostic to any distribution variations of the input data pixel intensity, enabling the reliable training of a DL model on input DRRs and applying the exact same model to conventional radiographs (or DRRs) during inference.