# Rare event detection by progressive clustering undersampling

## TL;DR

This paper introduces a new method called Progressive Clustering Undersampling to better detect rare events in datasets where one class is much more common than the other.

## Contribution

The novel Progressive Clustering Undersampling (PCU) method improves rare event detection by removing distant negative instances.

## Key findings

- PCU outperforms eight undersampling and two oversampling techniques on imbalanced datasets.
- The method identifies clusters with the highest concentration of positive instances for anomaly detection.
- PCU produces two optimized outputs for high F1-score and high precision.

## Abstract

Capturing rare events in severely imbalanced datasets is challenging, as the learning and optimization processes are often biased toward the majority class. To address this issue, this study explores various resampling techniques and introduces a novel method called Progressive Clustering Undersampling (PCU). This technique removes negative instances that are distant from positive ones. PCU was compared with eight common undersampling and two oversampling techniques, consistently outperforming them on highly imbalanced and noisy datasets.

The workflow demonstrates that rare anomalies can be effectively predicted using unsupervised methods based on frequency-driven decision boundaries. Progressive clustering ultimately identifies clusters with the highest concentration of positive instances. These delineated clusters are then saved by supervised models and used in the preparatory phase before prediction. The proposed method produces two outputs: one optimized for a high F1-score and the other for high precision. Overall, this approach presents a promising solution for identifying rare anomalies in complex, imbalanced data environments.