Generative deep learning for foundational video translation in ultrasound

TL;DR

This paper introduces a generative deep learning method for ultrasound video translation between CFD and greyscale modalities, producing highly realistic synthetic videos that enhance dataset balance and utility across clinical domains.

Contribution

A novel generative approach for ultrasound CFD-greyscale video translation using adversarial and perceptual losses, trained on large datasets, with demonstrated high realism and cross-domain applicability.

Findings

Synthetic videos achieved SSIM of 0.91+/-0.04, comparable to real videos.

Synthetic videos performed indistinguishably in classification and segmentation tasks.

Clinicians could not reliably distinguish real from synthetic videos, with accuracy around 54%.

Abstract

Deep learning (DL) has the potential to revolutionize image acquisition and interpretation across medicine, however, attention to data imbalance and missingness is required. Ultrasound data presents a particular challenge because in addition to different views and structures, it includes several sub-modalities-such as greyscale and color flow doppler (CFD)-that are often imbalanced in clinical studies. Image translation can help balance datasets but is challenging for ultrasound sub-modalities to date. Here, we present a generative method for ultrasound CFD-greyscale video translation, trained on 54,975 videos and tested on 8,368. The method developed leveraged pixel-wise, adversarial, and perceptual loses and utilized two networks: one for reconstructing anatomic structures and one for denoising to achieve realistic ultrasound imaging. Average pairwise SSIM between synthetic videos and ground truth was 0.91+/-0.04. Synthetic videos performed indistinguishably from real ones in DL classification and segmentation tasks and when evaluated by blinded clinical experts: F1 score was 0.9 for real and 0.89 for synthetic videos; Dice score between real and synthetic segmentation was 0.97. Overall clinician accuracy in distinguishing real vs synthetic videos was 54+/-6% (42-61%), indicating realistic synthetic videos. Although trained only on heart videos, the model worked well on ultrasound spanning several clinical domains (average SSIM 0.91+/-0.05), demonstrating foundational abilities. Together, these data expand the utility of retrospectively collected imaging and augment the dataset design toolbox for medical imaging.