Evaluating MCC for Low-Frequency Cyberattack Detection in Imbalanced Intrusion Detection Data

TL;DR

This paper demonstrates that Matthews Correlation Coefficient (MCC) offers a more reliable evaluation metric than accuracy for detecting low-frequency cyberattacks in imbalanced intrusion detection datasets, highlighting the importance of imbalance-aware assessment.

Contribution

The study compares accuracy and MCC for evaluating classifiers on imbalanced intrusion detection data, advocating MCC as a more trustworthy metric for low-frequency attack detection.

Findings

MCC provides a more accurate performance measure than accuracy for imbalanced data.

Meta-classifiers like LogitBoost and AdaBoost perform better in minority attack detection when evaluated with MCC.

Accuracy tends to overstate classifier performance in low-traffic attack scenarios.

Abstract

In many real-world network environments, several types of cyberattacks occur at very low rates compared to benign traffic, making them difficult for intrusion detection systems (IDS) to detect reliably. This imbalance causes traditional evaluation metrics, such as accuracy, to often overstate model performance in these conditions, masking failures on minority attack classes that are most important in practice. In this paper, we evaluate a set of base and meta classifiers on low-traffic attacks in the CSE-CIC-IDS2017 dataset and compare their reliability in terms of accuracy and Matthews Correlation Coefficient (MCC). The results show that accuracy consistently inflates performance, while MCC provides a more accurate assessment of a classifier's performance across both majority and minority classes. Meta-classification methods, such as LogitBoost and AdaBoost, demonstrate more effective minority class detection when measured by MCC, revealing trends that accuracy fails to capture. These findings establish the need for imbalance-aware evaluation and make MCC a more trustworthy metric for IDS research involving low-traffic cyberattacks.