Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning

TL;DR

This paper presents MoGLo-Net, a deep learning network with self-attention for improved 3D reconstruction in handheld photoacoustic and ultrasound imaging, enabling detailed visualization of complex structures without external sensors.

Contribution

Introduction of MoGLo-Net, a novel motion-based learning network with self-attention and correlation modules for accurate 3D reconstruction in PAUS imaging, including Doppler and photoacoustic modalities.

Findings

MoGLo-Net outperforms existing methods in accuracy and quality.

Effective motion estimation without external sensors.

Enables 3D visualization of vasculature in photoacoustic imaging.

Abstract

This study introduces a motion-based learning network with a global-local self-attention module (MoGLo-Net) to enhance 3D reconstruction in handheld photoacoustic and ultrasound (PAUS) imaging. Standard PAUS imaging is often limited by a narrow field of view and the inability to effectively visualize complex 3D structures. The 3D freehand technique, which aligns sequential 2D images for 3D reconstruction, faces significant challenges in accurate motion estimation without relying on external positional sensors. MoGLo-Net addresses these limitations through an innovative adaptation of the self-attention mechanism, which effectively exploits the critical regions, such as fully-developed speckle area or high-echogenic tissue area within successive ultrasound images to accurately estimate motion parameters. This facilitates the extraction of intricate features from individual frames. Additionally, we designed a patch-wise correlation operation to generate a correlation volume that is highly correlated with the scanning motion. A custom loss function was also developed to ensure robust learning with minimized bias, leveraging the characteristics of the motion parameters. Experimental evaluations demonstrated that MoGLo-Net surpasses current state-of-the-art methods in both quantitative and qualitative performance metrics. Furthermore, we expanded the application of 3D reconstruction technology beyond simple B-mode ultrasound volumes to incorporate Doppler ultrasound and photoacoustic imaging, enabling 3D visualization of vasculature. The source code for this study is publicly available at: https://github.com/guhong3648/US3D