Fast Unsupervised Deep Outlier Model Selection with Hypernetworks
Xueying Ding, Yue Zhao, Leman Akoglu
TL;DR
This paper introduces HYPER, a hypernetwork-based method for unsupervised deep outlier detection that efficiently tunes hyperparameters and models without labeled data, significantly improving speed and performance across multiple tasks.
Contribution
HYPER is a novel hypernetwork approach that enables fast, unsupervised hyperparameter tuning and model selection for deep outlier detection, addressing validation and search challenges.
Findings
HYPER outperforms 8 baselines on 35 OD tasks.
It achieves significant efficiency gains in hyperparameter search.
HYPER maintains high detection performance without labeled validation data.
Abstract
Outlier detection (OD) finds many applications with a rich literature of numerous techniques. Deep neural network based OD (DOD) has seen a recent surge of attention thanks to the many advances in deep learning. In this paper, we consider a critical-yet-understudied challenge with unsupervised DOD, that is, effective hyperparameter (HP) tuning/model selection. While several prior work report the sensitivity of OD models to HPs, it becomes ever so critical for the modern DOD models that exhibit a long list of HPs. We introduce HYPER for tuning DOD models, tackling two fundamental challenges: (1) validation without supervision (due to lack of labeled anomalies), and (2) efficient search of the HP/model space (due to exponential growth in the number of HPs). A key idea is to design and train a novel hypernetwork (HN) that maps HPs onto optimal weights of the main DOD model. In turn, HYPER…
Peer Reviews
Decision·ICLR 2024 Conference Withdrawn Submission
- The problem setup is realistic and important, as there are outliers in real-data scenarios. Moreover, its detection helps to improve the modeling. - The approach seems to improve performance and inference speed.
- The manuscript is very difficult to read and it is confusing in some parts, especially when explaining the proposed method. - Due to the lack of effective explanation, the authors consume 7 out of 9 pages introducing the method and just dedicate 2 pages to the experiments. Given that this is a paper with empirical methods, it should dedicate more space elaborating on experiments and results. - There is no clear explanation of the meta-learning algorithm, which is a very important stage in this
1. The paper introduces a novel approach to effectively tune hyperparameters and select models for deep neural network based outlier detection. The use of a hypernetwork to dynamically generate weights for many DOD models is a unique and innovative idea. 2. The authors tackle key challenges in unsupervised DOD, including validation without supervision and efficient search of the hyperparameter/model space. By addressing these challenges, HYPER achieves significant performance improvements and s
1. The authors do not compare their framework to methods that use labeled data. It is unclear how their framework would perform on datasets with labeled data, which is the setting in which outlier detection is typically applied. 2. The authors only evaluate their framework on a relatively small set of benchmark datasets. It is possible that their framework would not generalize well to other datasets. While the authors demonstrate the effectiveness of HYPER on benchmark datasets, there is no ev
1. The paper introduces an interesting setting. 2. Method gives nice numerical results.
1. Notation HP and HN is misleading. Maybe is better to use full names. 2. It is hard to understand the setting of the model. In OD we do not have labels. But in meta training on historical data, we use labels. 3. Figure 2 is not clear, especially part 3.2 4. Experiments work with fully connected AutoEncoder (AE) for DOD on tabular data. The architecture is simple, and the datasets are also. 5. Authors should use image datasets. 6. Authors should present results on convolution layers. 7. Author
1. The paper studies a well-motivated and important problem. In outlier detection, the labeled outliers are often not available. The performance of deep outlier detection models usually depends on the selected hyperparameters. UDOMS is an important problem in practice. 2. The paper provides an extensive empirical evaluation to showcase the efficacy of the proposed method. The proposed method achieves the lowest average ROC ranking on 35 tabular datasets.
1. In comparison to existing works, e.g., MetaOD, the major contribution of this paper is learning hypernetworks for saving individual model training time. From Fig. 4 we can see that MetaOD is actually faster than the proposed method. The training of hypernetworks becomes more complicated and involves new hyperparameters. 2. The whole framework involves several learnable components, e.g., data embedding, model embedding, performance estimator, and the hypernetworks. The final UDOMS performance
1. A large set of experiments was conducted, involving comparisons with 8 baselines across 35 datasets.
1. Utilizing the test set during training is not a valid practice in machine learning. 2. The majority of the datasets and networks (3-layer convolutional networks) employed in the study are relatively small, raising questions about the scalability and practicality of the proposed method when applied to larger datasets. 3. The presentation and structure of this paper significantly hinder its readability, to the point where I find it challenging to proceed with the review. I recommend that the au
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAnomaly Detection Techniques and Applications
MethodsHyperNetwork