Bayesian Active Learning By Distribution Disagreement
Thorben Werner, Lars Schmidt-Thieme
TL;DR
This paper introduces BALSA, a novel active learning algorithm for regression using normalizing flows, which effectively distinguishes between types of uncertainty and achieves state-of-the-art results across multiple datasets.
Contribution
The paper develops BALSA, an adaptation of BALD for regression with normalizing flows, enhancing uncertainty quantification and active learning performance.
Findings
BALSA outperforms existing methods on four datasets.
Normalizing flows enable direct uncertainty estimation in regression.
Sophisticated algorithms are needed for effective pool-based active learning.
Abstract
Active Learning (AL) for regression has been systematically under-researched due to the increased difficulty of measuring uncertainty in regression models. Since normalizing flows offer a full predictive distribution instead of a point forecast, they facilitate direct usage of known heuristics for AL like Entropy or Least-Confident sampling. However, we show that most of these heuristics do not work well for normalizing flows in pool-based AL and we need more sophisticated algorithms to distinguish between aleatoric and epistemic uncertainty. In this work we propose BALSA, an adaptation of the BALD algorithm, tailored for regression with normalizing flows. With this work we extend current research on uncertainty quantification with normalizing flows \cite{berry2023normalizing, berry2023escaping} to real world data and pool-based AL with multiple acquisition functions and query sizes. We…
Peer Reviews
Decision·Submitted to ICLR 2025
The paper is original in its focus on developing active learning strategies specifically for regression with normalizing flows, while much of the AL research traditionally focuses on classification tasks. The paper offers a comprehensive benchmark for AL in regression with predictive distributions. The experiments are robust, testing BALSA across four diverse regression datasets and two model architectures. This extensive comparison proves BALSA's effectiveness and generalizability.
The presentation of this paper could be improved. For example, Figure 1 is presented without reference or explanation in the text, which reduces clarity for the reader. In Pair Comparison, the paper introduces a pairwise approach to approximate Eq. 2, which is one of the core components of BALSA. However, the paper does not adequately explain why this approach is effective, nor does it discuss any potential trade-offs or advantages that led to this specific choice. The paper claims that BA
The authors report strong results on the chosen datasets.
The writing is very poor and hard to follow. The notation exhibits a lack of rigor and mathematical expressions are not properly introduced. For example, it is unclear what each \theta_i in eq. (1) refers to. It's unclear how these quantities are measured. 'BALD' is also mentioned many times and never properly described. While the authors claim that the datasets were selected to 'provide maximal intersection with other literature for AL with regression', they are quite small by modern standar
1. The paper addresses an under-researched area of the literature. 2. The broader approach to the BALD acquisition criterion is interesting and appears to have strong potential.
1. Formatting issues: - Ensure paragraphs are indented throughout the document. - Correct the initial quotation marks by using `` for opening quotes in LaTeX. - In Figure 3, the abundance of methods makes the lines difficult to differentiate. Consider moving some of these methods to the appendix to enhance readability. 2. Results presentation: The results lack standard deviation and confidence intervals, making it challenging to fully trust the conclusions. Including these would provid
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms
MethodsNormalizing Flows