Cross Modality 3D Navigation Using Reinforcement Learning and Neural Style Transfer

TL;DR

This paper introduces a novel framework combining Multi-Agent Reinforcement Learning and Neural Style Transfer to enable cross-modality 3D navigation in medical imaging, especially useful for modalities like ultrasound, without requiring labeled clinical data.

Contribution

It presents a new approach that leverages style transfer to generate synthetic training environments and conditions agents on 2D slices for 3D guidance across different imaging modalities.

Findings

Agents successfully navigate in synthetic CT environments.

Framework generalizes to clinical CT volumes without labeled data.

Enables 3D guidance in ultrasound imaging.

Abstract

This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imaging modalities, such as ultrasound imaging. This is an important step towards user guidance during the acquisition of standardised diagnostic view planes, improving diagnostic consistency and facilitating better case comparison.