A Unified Approach Towards Active Learning and Out-of-Distribution Detection
Sebastian Schmidt, Leonard Schenk, Leo Schwinn, Stephan G\"unnemann
TL;DR
This paper introduces SISOM, a unified method that effectively combines active learning and out-of-distribution detection using feature space distances, achieving top performance in multiple benchmarks.
Contribution
SISOM is the first unified approach that addresses both active learning and OOD detection simultaneously using feature space metrics.
Findings
Achieved first place in two OpenOOD benchmarks.
Secured second place in one OpenOOD benchmark.
Outperformed existing methods in active learning benchmarks.
Abstract
When applying deep learning models in open-world scenarios, active learning (AL) strategies are crucial for identifying label candidates from a nearly infinite amount of unlabeled data. In this context, robust out-of-distribution (OOD) detection mechanisms are essential for handling data outside the target distribution of the application. However, current works investigate both problems separately. In this work, we introduce SISOM as the first unified solution for both AL and OOD detection. By leveraging feature space distance metrics SISOM combines the strengths of the currently independent tasks to solve both effectively. We conduct extensive experiments showing the problems arising when migrating between both tasks. In these evaluations SISOM underlined its effectiveness by achieving first place in two of the widely used OpenOOD benchmarks and second place in the remaining one. In…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. This paper proposes SISOM, the first approach designed to jointly solve OOD detection and AL. 2. The experiments on AL and OOD detection are extensive.
1. In the active learning (AL) experiments, does the unlabeled data include out-of-distribution (OOD) samples? It appears that AL and OOD are discussed as separate scenarios in the experiments. I would like to know if the proposed method can work in a coupled scenario where the unlabeled data contains some OOD samples as noisy data. 2. The improvement in experimental performance is limited. 3. In AL experiments, if 5,000 labeled samples are added in each AL cycle, what is the total number of la
- This study is significant in that it explores the practical applicability of deep learning by simultaneously considering both AL and OoDD tasks. The proposed method is particularly valuable as it is hyperparameter-free, which greatly enhances its practical usability. - The paper is clearly written and easy to follow. Additionally, the paper includes comparative experiments with many well-known methods in both AL and OoDD tasks.
- I have concerns about certain overclaims in the study. First, the authors state that they propose the "first unified solution for both AL and OoDD." I understand that this claim is based on the fact that the proposed SISOM score, using a distance ratio, can be applied to both AL and OoDD. However, existing uncertainty metrics, such as MSP and MC Dropout, can also be used as a single metric for both AL and OoDD, making it difficult to assert that this study is the first unified solution. Second
1) This paper tries to study a unified approach to address active learning and out-of-distribution detection simultaneously. This problem has rarely been studied. 2) A new distance metrics has been introduced to help improve both the performance of active learning and OOD detection.
1) Why it is necessary to unify active learning and OOD detection into one framework. How can these tasks help each other? More explanations or theoretical analyses are needed. The contribution of this work to the field of active learning and OOD detection needs to be emphasized. 2) Active learning aims to exploit unlabeled data. Actually, in the field of semi-supervised learning, various works are focusing on open-set or open-world semi-supervised learning, i.e., the unlabeled data contains out
1. The paper provides a clear and detailed description of the SISOM method, making it accessible and understandable. 2. The authors conducted extensive experiments across various datasets and compared the method against multiple benchmarks, substantiating SISOM's efficacy.
1. The problem addressed in this paper is essentially consistent with existing open-set active learning (OSAL) methods [1-7]. However, the authors provide a highly inappropriate justification for separating the two, allowing their proposed method to avoid comparison with these existing methods. In OSAL tasks, the goal of AL is to query known class samples with high information content. The authors' assertion that "existing methods address AL and OOD separately" is misleading, as it merely repres
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms · Fault Detection and Control Systems