Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Loss

TL;DR

This paper introduces a self-supervised ultrasound video segmentation method using feature prediction and a novel 3D localised loss, significantly improving performance with limited annotated data.

Contribution

It adapts V-JEPA for ultrasound videos and proposes a 3D localisation auxiliary task to enhance ViT-based model performance on small datasets.

Findings

Up to 8.35% segmentation improvement with only 10% of training data.

Significant gains across various frozen encoder configurations.

First application of V-JEPA to ultrasound video segmentation.

Abstract

Acquiring and annotating large datasets in ultrasound imaging is challenging due to low contrast, high noise, and susceptibility to artefacts. This process requires significant time and clinical expertise. Self-supervised learning (SSL) offers a promising solution by leveraging unlabelled data to learn useful representations, enabling improved segmentation performance when annotated data is limited. Recent state-of-the-art developments in SSL for video data include V-JEPA, a framework solely based on feature prediction, avoiding pixel level reconstruction or negative samples. We hypothesise that V-JEPA is well-suited to ultrasound imaging, as it is less sensitive to noisy pixel-level detail while effectively leveraging temporal information. To the best of our knowledge, this is the first study to adopt V-JEPA for ultrasound video data. Similar to other patch-based masking SSL techniques such as VideoMAE, V-JEPA is well-suited to ViT-based models. However, ViTs can underperform on small medical datasets due to lack of inductive biases, limited spatial locality and absence of hierarchical feature learning. To improve locality understanding, we propose a novel 3D localisation auxiliary task to improve locality in ViT representations during V-JEPA pre-training. Our results show V-JEPA with our auxiliary task improves segmentation performance significantly across various frozen encoder configurations, with gains up to 3.4\% using 100\% and up to 8.35\% using only 10\% of the training data.