Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysis

TL;DR

This paper introduces an information-theoretic framework for pool-based active learning, modeling it as noisy lossy compression to derive fundamental limits on label complexity and generalization error.

Contribution

It reformulates pool-based active learning as a noisy lossy compression problem and derives new theoretical bounds using finite blocklength analysis.

Findings

Derived lower bounds on label complexity and generalization error.

Identified the impact of overfitting and inductive bias mismatch.

Connected active learning limits to established information-theoretic principles.

Abstract

This paper proposes an information-theoretic framework for analyzing the theoretical limits of pool-based active learning (AL), in which a subset of instances is selectively labeled. The proposed framework reformulates pool-based AL as a noisy lossy compression problem by mapping pool observations to noisy symbol observations, data selection to compression, and learning to decoding. This correspondence enables a unified information-theoretic analysis of data selection and learning in pool-based AL. Applying finite blocklength analysis of noisy lossy compression, we derive information-theoretic lower bounds on label complexity and generalization error that serve as theoretical limits for a given learning algorithm under its associated optimal data selection strategy. Specifically, our bounds include terms that reflect overfitting induced by the learning algorithm and the discrepancy between its inductive bias and the target task, and are closely related to established information-theoretic bounds and stability theory, which have not been previously applied to the analysis of pool-based AL. These properties yield a new theoretical perspective on pool-based AL.