When Imbalance Comes Twice: Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation

TL;DR

This paper investigates how class imbalance and label shift affect active learning strategies in binary semantic segmentation, demonstrating that certain strategies remain effective despite these challenges.

Contribution

It introduces a simulation framework to study the impact of class imbalance and label shift on active learning, comparing standard strategies under controlled conditions.

Findings

Active learning strategies remain effective under high class imbalance.

Entropy-based and core-set strategies outperform random sampling.

Label shift reduces the efficiency of active learning methods.

Abstract

The aim of Active Learning is to select the most informative samples from an unlabelled set of data. This is useful in cases where the amount of data is large and labelling is expensive, such as in machine vision or medical imaging. Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images. This results, on the one hand, in a strong class imbalance in defect distribution and, on the other hand, in a potential label shift caused by limited storage. To understand how these two forms of imbalance affect active learning algorithms, we propose a simulation study based on two open-source datasets. We artificially create datasets for which we control the levels of class imbalance and label shift. Three standard active learning selection strategies are compared: random sampling, entropy-based selection, and core-set selection. We demonstrate that active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets. We also illustrate and measure the loss of efficiency that occurs in the situation a strong label shift.