COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentation

TL;DR

This paper introduces COLosSAL, a comprehensive benchmark for evaluating cold-start active learning strategies in 3D medical image segmentation, highlighting the challenges and current gaps in the field.

Contribution

We present a new benchmark for cold-start active learning in 3D medical segmentation, evaluating six strategies across five datasets and analyzing their performance and open questions.

Findings

Cold-start AL remains an unsolved challenge for 3D segmentation.

Performance varies significantly depending on the strategy and budget.

Some trends and insights are identified to guide future research.

Abstract

Medical image segmentation is a critical task in medical image analysis. In recent years, deep learning based approaches have shown exceptional performance when trained on a fully-annotated dataset. However, data annotation is often a significant bottleneck, especially for 3D medical images. Active learning (AL) is a promising solution for efficient annotation but requires an initial set of labeled samples to start active selection. When the entire data pool is unlabeled, how do we select the samples to annotate as our initial set? This is also known as the cold-start AL, which permits only one chance to request annotations from experts without access to previously annotated data. Cold-start AL is highly relevant in many practical scenarios but has been under-explored, especially for 3D medical segmentation tasks requiring substantial annotation effort. In this paper, we present a benchmark named COLosSAL by evaluating six cold-start AL strategies on five 3D medical image segmentation tasks from the public Medical Segmentation Decathlon collection. We perform a thorough performance analysis and explore important open questions for cold-start AL, such as the impact of budget on different strategies. Our results show that cold-start AL is still an unsolved problem for 3D segmentation tasks but some important trends have been observed. The code repository, data partitions, and baseline results for the complete benchmark are publicly available at https://github.com/MedICL-VU/COLosSAL.