Direct Acquisition Optimization for Low-Budget Active Learning
Zhuokai Zhao, Yibo Jiang, Yuxin Chen
TL;DR
This paper introduces Direct Acquisition Optimization (DAO), a novel active learning algorithm designed to improve sample selection accuracy in low-budget scenarios by optimizing expected true loss reduction, outperforming existing methods.
Contribution
The paper proposes DAO, a new active learning method that uses influence functions and bias mitigation to enhance low-budget sample selection without heavy computation.
Findings
DAO outperforms state-of-the-art methods in low-budget settings
Empirical results on seven benchmarks demonstrate DAO's effectiveness
DAO achieves more accurate error reduction estimation
Abstract
Active Learning (AL) has gained prominence in integrating data-intensive machine learning (ML) models into domains with limited labeled data. However, its effectiveness diminishes significantly when the labeling budget is low. In this paper, we first empirically observe the performance degradation of existing AL algorithms in the low-budget settings, and then introduce Direct Acquisition Optimization (DAO), a novel AL algorithm that optimizes sample selections based on expected true loss reduction. Specifically, DAO utilizes influence functions to update model parameters and incorporates an additional acquisition strategy to mitigate bias in loss estimation. This approach facilitates a more accurate estimation of the overall error reduction, without extensive computations or reliance on labeled data. Experiments demonstrate DAO's effectiveness in low budget settings, outperforming…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
The study addresses notable limitations in existing EER methods, potentially enhancing their applicability within deep learning frameworks. Of particular interest is the use of influence functions to approximate model parameter updates, thereby alleviating the need for repeated model retraining. The experiments are well-designed, covering a broad range of popular benchmark datasets. The results are robust and persuasive, demonstrating performance gains over existing active learning methods.
The main concern lies in the limited technical contributions of the methodology, as the primary techniques appear to have been adapted from existing studies. Besides, although influence functions offer an efficient approximation for parameter updates, they can be both memory and computationally intensive, especially with deep models. This may limit the practicality of applying the proposed method to deep models. A minor point is that the authors claim to use a surrogate model to estimate labels
The paper conducts experiment across a large number of datasets, showing the consistent improvement in accuracy over baseline methods. The paper also draws from multiple latest advances in deep active learning, and effectively integrated them together to ensure better performance and theoretical justification.
1. The paper does not compare against TypiClust [1], which is known to outperform all the baseline strategies here in low budget settings. 2. In the high budget experiments, the total budget is not sufficiently large, since each class only has 50 annotation budget on average. It would be interesting to see the algorithm's performance with even larger annotation budget. 3. The notation $\ell(x, f)$ is very confusing. The loss function should be a function of the label $y$ as well. This seems to h
Originality: This paper focus on extreme low budge setting while using expected loss reduction as criterion for sample selection. Most of active learning are based on heuristics, which emphasize the computational efficiency in sample selection process. But the training objective is different from the heuristic for sample selection, making them less effective. However, evaluating expected loss on test data requires some labeled validation set. This is not suitable for active learning with low bu
Contribution: This paper focus on active learning with low budge setting and the algorithm presented solves three issues: 1) how to obtain pseudo-labels in unlabeled data; 2) how to approximate new model parameter without retraining; 3) how provide low bias estimation on true expected loss on unlabled data. The issue for 1) is that the obtaining a good proposed surrogate function is difficult.This model plays a central role for the success of this algorithm both theoretically and practically
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExperimental Learning in Engineering · Machine Learning and Algorithms · Analog and Mixed-Signal Circuit Design