On the reusability of samples in active learning

TL;DR

This paper investigates the limits of sample reusability in active learning, demonstrating that universal reusability is impossible due to inherent undersampling, but identifying conditions where reusability can occur.

Contribution

It provides a theoretical and empirical analysis of sample reusability in active learning, highlighting its limitations and potential conditions for reusability between classifiers.

Findings

Universal reusability in active learning does not exist.

Reusability depends on dataset and classifier pairs.

Importance-weighted active learning impacts reusability.

Abstract

An interesting but not extensively studied question in active learning is that of sample reusability: to what extent can samples selected for one learner be reused by another? This paper explains why sample reusability is of practical interest, why reusability can be a problem, how reusability could be improved by importance-weighted active learning, and which obstacles to universal reusability remain. With theoretical arguments and practical demonstrations, this paper argues that universal reusability is impossible. Because every active learning strategy must undersample some areas of the sample space, learners that depend on the samples in those areas will learn more from a random sample selection. This paper describes several experiments with importance-weighted active learning that show the impact of the reusability problem in practice. The experiments confirmed that universal reusability does not exist, although in some cases -- on some datasets and with some pairs of classifiers -- there is sample reusability. Finally, this paper explores the conditions that could guarantee the reusability between two classifiers.