Accurate and Reliable Methods for 5G UAV Jamming Identification With Calibrated Uncertainty

TL;DR

This paper introduces five combined methods to enhance accuracy and reliability of DNN-based 5G UAV jamming detection, emphasizing calibrated uncertainty and comparing multiple metrics for optimal performance.

Contribution

It proposes novel hybrid preprocessing and post-processing techniques that improve both accuracy and reliability in UAV security classification tasks.

Findings

Hybrid methods outperform standalone DNN softmax classification.

At least one method achieves better reliability scores than XGB classifier.

Calibrated uncertainty improves decision-making in UAV jamming detection.

Abstract

Only increasing accuracy without considering uncertainty may negatively impact Deep Neural Network (DNN) decision-making and decrease its reliability. This paper proposes five combined preprocessing and post-processing methods for time-series binary classification problems that simultaneously increase the accuracy and reliability of DNN outputs applied in a 5G UAV security dataset. These techniques use DNN outputs as input parameters and process them in different ways. Two methods use a well-known Machine Learning (ML) algorithm as a complement, and the other three use only confidence values that the DNN estimates. We compare seven different metrics, such as the Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Mean Confidence (MC), Mean Accuracy (MA), Normalized Negative Log Likelihood (NLL), Brier Score Loss (BSL), and Reliability Score (RS) and the tradeoffs between them to evaluate the proposed hybrid algorithms. First, we show that the eXtreme Gradient Boosting (XGB) classifier might not be reliable for binary classification under the conditions this work presents. Second, we demonstrate that at least one of the potential methods can achieve better results than the classification in the DNN softmax layer. Finally, we show that the prospective methods may improve accuracy and reliability with better uncertainty calibration based on the assumption that the RS determines the difference between MC and MA metrics, and this difference should be zero to increase reliability. For example, Method 3 presents the best RS of 0.65 even when compared to the XGB classifier, which achieves RS of 7.22.