Is $F_1$ Score Suboptimal for Cybersecurity Models? Introducing $C_{score}$, a Cost-Aware Alternative for Model Assessment

TL;DR

This paper introduces $C_{score}$, a cost-aware evaluation metric for cybersecurity models that accounts for different error costs, outperforming $F_1$ score in cost savings.

Contribution

It proposes a novel cost-aware metric, $C_{score}$, that incorporates error costs into model evaluation, replacing the traditional $F_1$ score.

Findings

$C_{score}$ achieves an average of 49% cost savings.

The metric effectively incorporates different false positive and false negative costs.

Application to five cybersecurity datasets demonstrates practical benefits.

Abstract

The cost of errors related to machine learning classifiers, namely, false positives and false negatives, are not equal and are application dependent. For example, in cybersecurity applications, the cost of not detecting an attack is very different from marking a benign activity as an attack. Various design choices during machine learning model building, such as hyperparameter tuning and model selection, allow a data scientist to trade-off between these two errors. However, most of the commonly used metrics to evaluate model quality, such as $F_1$ score, which is defined in terms of model precision and recall, treat both these errors equally, making it difficult for users to optimize for the actual cost of these errors. In this paper, we propose a new cost-aware metric, $C_{score}$ based on precision and recall that can replace $F_1$ score for model evaluation and selection. It includes a cost ratio that takes into account the differing costs of handling false positives and false negatives. We derive and characterize the new cost metric, and compare it to $F_1$ score. Further, we use this metric for model thresholding for five cybersecurity related datasets for multiple cost ratios. The results show an average cost savings of 49%.