Am I Rare? An Intelligent Summarization Approach for Identifying Hidden Anomalies

TL;DR

This paper introduces INSIDENT, an intelligent clustering-based summarization method that preserves data distribution to effectively identify hidden anomalies in network traffic, reducing computational costs while maintaining detection accuracy.

Contribution

The paper presents a novel clustering-based summarization approach that maintains original data distribution, enhancing anomaly detection effectiveness in summarized network traffic data.

Findings

Summarized data can substitute original data in anomaly detection.

INSIDENT preserves data distribution and improves anomaly detection accuracy.

Experimental results on benchmark datasets validate the approach's effectiveness.

Abstract

Monitoring network traffic data to detect any hidden patterns of anomalies is a challenging and time-consuming task that requires high computing resources. To this end, an appropriate summarization technique is of great importance, where it can be a substitute for the original data. However, the summarized data is under the threat of removing anomalies. Therefore, it is vital to create a summary that can reflect the same pattern as the original data. Therefore, in this paper, we propose an INtelligent Summarization approach for IDENTifying hidden anomalies, called INSIDENT. The proposed approach guarantees to keep the original data distribution in summarized data. Our approach is a clustering-based algorithm that dynamically maps original feature space to a new feature space by locally weighting features in each cluster. Therefore, in new feature space, similar samples are closer, and consequently, outliers are more detectable. Besides, selecting representatives based on cluster size keeps the same distribution as the original data in summarized data. INSIDENT can be used both as the preprocess approach before performing anomaly detection algorithms and anomaly detection algorithm. The experimental results on benchmark datasets prove a summary of the data can be a substitute for original data in the anomaly detection task.