A Mutual Information Lower Bound for Multimodal Regression Active Learning

TL;DR

This paper introduces a mutual information lower bound for active learning in multimodal regression, effectively capturing epistemic uncertainty and outperforming existing methods on benchmarks.

Contribution

It proposes a Two-Index framework and a practical MI-LB acquisition function for better active learning in multimodal regression tasks.

Findings

MI-LB matches or exceeds baseline performance on benchmarks

It effectively captures epistemic uncertainty in multimodal systems

Outperforms geometric and Fisher-based baselines in diverse scenarios

Abstract

Active learning for continuous regression has lacked an acquisition function that targets epistemic uncertainty when the predictive distribution is multimodal: variance misses modal disagreement, and information-theoretic targets like BALD are designed for discrete outputs. We introduce a Two-Index framework that makes this separation explicit: one stochastic index selects among competing model hypotheses (epistemic source), while a second governs within-hypothesis randomness (aleatoric source). An entropy decomposition within the framework identifies the mutual information between the output and the epistemic index as a principled acquisition objective, and we prove this quantity vanishes as the model is trained on growing datasets, confirming that it captures exactly the uncertainty data can resolve. Because this mutual information is intractable for continuous outputs, we derive the Mutual Information Lower Bound (MI-LB) acquisition function, a closed-form approximation for Mixture Density Network ensembles. On benchmarks featuring multimodal systems, MI-LB matches or beats every baseline evaluated and is the only method to do so consistently -- geometric and Fisher-based baselines compete only when the input space already encodes the multimodality, and collapse otherwise.