# Evaluation of Confusion Behaviors in SEI Models

## TL;DR

This paper examines how RFML models for SEI tasks fail under different conditions, finding that they often misclassify signals into a small group of classes when signal quality is low.

## Contribution

The study introduces a systematic analysis of confusion behaviors in SEI models across varying SNR and training data quantities.

## Key findings

- RFML models tend to misclassify signals into a small subset of classes (≈10%) at low SNR.
- Ensemble models are less brittle at low SNR but not the highest-performing at high SNR.
- Error patterns are consistent across different SEI model architectures.

## Abstract

Radio Frequency Machine Learning (RFML) has in recent years become a popular method for performing a variety of classification tasks on received signals. Among these tasks is Specific Emitter Identification (SEI), which seeks to associate a received signal with the physical emitter that transmitted it. Many different model architectures, including individual classifiers and ensemble methods, have proven their capabilities for producing high accuracy classification results when performing SEI. Though the works studying different model architectures report on successes, there is a notable absence regarding the examination of systemic failures and negative traits associated with learned behaviors. This work studies those failure patterns for a 64-radio SEI classification problem by isolating common patterns in incorrect classification results across multiple model architectures and two distinct control variables: Signal-to-Noise Ratio (SNR) and the quantity of training data utilized. This work finds that many of the RFML-based models devolve to selecting from amongst a small subset of classes (≈10% of classes) as SNRs decrease and that observed errors are reasonably consistent across different SEI models and architectures. Moreover, our results validate the expectation that ensemble models are generally less brittle, particularly at a low SNR, yet they appear not to be the highest-performing option at a high SNR.