US-X Complete: A Multi-Modal Approach to Anatomical 3D Shape Recovery

TL;DR

This paper introduces a multi-modal deep learning approach that combines ultrasound and X-ray images to improve 3D vertebral reconstruction, overcoming ultrasound's limitations in visualizing complete vertebral anatomy.

Contribution

The novel method leverages paired ultrasound and X-ray data for enhanced 3D vertebral completion, enabling more accurate spine visualization without registration to preoperative scans.

Findings

Significant improvement in vertebral reconstruction accuracy (p < 0.001).

Successful phantom studies demonstrating clinical potential.

Complete lumbar spine visualization overlayed on ultrasound.

Abstract

Ultrasound offers a radiation-free, cost-effective solution for real-time visualization of spinal landmarks, paraspinal soft tissues and neurovascular structures, making it valuable for intraoperative guidance during spinal procedures. However, ultrasound suffers from inherent limitations in visualizing complete vertebral anatomy, in particular vertebral bodies, due to acoustic shadowing effects caused by bone. In this work, we present a novel multi-modal deep learning method for completing occluded anatomical structures in 3D ultrasound by leveraging complementary information from a single X-ray image. To enable training, we generate paired training data consisting of: (1) 2D lateral vertebral views that simulate X-ray scans, and (2) 3D partial vertebrae representations that mimic the limited visibility and occlusions encountered during ultrasound spine imaging. Our method integrates morphological information from both imaging modalities and demonstrates significant improvements in vertebral reconstruction (p < 0.001) compared to state of art in 3D ultrasound vertebral completion. We perform phantom studies as an initial step to future clinical translation, and achieve a more accurate, complete volumetric lumbar spine visualization overlayed on the ultrasound scan without the need for registration with preoperative modalities such as computed tomography. This demonstrates that integrating a single X-ray projection mitigates ultrasound's key limitation while preserving its strengths as the primary imaging modality. Code and data can be found at https://github.com/miruna20/US-X-Complete