An Experiment on Feature Selection using Logistic Regression

TL;DR

This study compares feature selection methods based on L1 and L2 regularization in logistic regression, demonstrating that their combined feature set performs similarly to individual methods across different models and metrics.

Contribution

It synthesizes L1 and L2 regularization results for feature selection and evaluates their effectiveness on complex models using a large cybersecurity dataset.

Findings

No significant accuracy difference between L1 and L2 feature selection methods.

Combined features from L1 and L2 yield comparable performance on complex models.

Standard metrics confirm similar effectiveness of the combined feature set.

Abstract

In supervised machine learning, feature selection plays a very important role by potentially enhancing explainability and performance as measured by computing time and accuracy-related metrics. In this paper, we investigate a method for feature selection based on the well-known L1 and L2 regularization strategies associated with logistic regression (LR). It is well known that the learned coefficients, which serve as weights, can be used to rank the features. Our approach is to synthesize the findings of L1 and L2 regularization. For our experiment, we chose the CIC-IDS2018 dataset owing partly to its size and also to the existence of two problematic classes that are hard to separate. We report first with the exclusion of one of them and then with its inclusion. We ranked features first with L1 and then with L2, and then compared logistic regression with L1 (LR+L1) against that with L2 (LR+L2) by varying the sizes of the feature sets for each of the two rankings. We found no significant difference in accuracy between the two methods once the feature set is selected. We chose a synthesis, i.e., only those features that were present in both the sets obtained from L1 and that from L2, and experimented with it on more complex models like Decision Tree and Random Forest and observed that the accuracy was very close in spite of the small size of the feature set. Additionally, we also report on the standard metrics: accuracy, precision, recall, and f1-score.