An explainable Recursive Feature Elimination to detect Advanced Persistent Threats using Random Forest classifier

TL;DR

This paper presents an explainable intrusion detection framework combining Recursive Feature Elimination and Random Forests, achieving high accuracy and interpretability for detecting Advanced Persistent Threats using the CICIDS2017 dataset.

Contribution

It introduces an explainable RF-RFE approach that improves APT detection accuracy and transparency over traditional methods.

Findings

Detection accuracy of 99.9% achieved

Reduced false positives and computational cost

Enhanced interpretability with SHAP explanations

Abstract

Intrusion Detection Systems (IDS) play a vital role in modern cybersecurity frameworks by providing a primary defense mechanism against sophisticated threat actors. In this paper, we propose an explainable intrusion detection framework that integrates Recursive Feature Elimination (RFE) with Random Forest (RF) to enhance detection of Advanced Persistent Threats (APTs). By using CICIDS2017 dataset, the approach begins with comprehensive data preprocessing and narrows down the most significant features via RFE. A Random Forest (RF) model was trained on the refined feature set, with SHapley Additive exPlanations (SHAP) used to interpret the contribution of each selected feature. Our experiment demonstrates that the explainable RF-RFE achieved a detection accuracy of 99.9%, reducing false positive and computational cost in comparison to traditional classifiers. The findings underscore the effectiveness of integrating explainable AI and feature selection to develop a robust, transparent, and deployable IDS solution.