The Importance of Being Earnest: Performance of Modulation Classification for Real RF Signals

TL;DR

This study evaluates digital modulation classification (DMC) performance in real-world wireless networks using a large RF dataset, revealing its high accuracy under certain conditions but also highlighting challenges in generalization and practical deployment.

Contribution

The paper provides the first extensive real-world evaluation of DMC with heterogeneous hardware and interference, emphasizing the importance of diverse training data and robust feature design.

Findings

DMC achieves high accuracy with comprehensive training data.

Performance drops significantly with mismatched environments.

Public RF dataset facilitates realistic DMC evaluation.

Abstract

Digital modulation classification (DMC) can be highly valuable for equipping radios with increased spectrum awareness in complex emerging wireless networks. However, as the existing literature is overwhelmingly based on theoretical or simulation results, it is unclear how well DMC performs in practice. In this paper we study the performance of DMC in real-world wireless networks, using an extensive RF signal dataset of 250,000 over-the-air transmissions with heterogeneous transceiver hardware and co-channel interference. Our results show that DMC can achieve a high classification accuracy even under the challenging real-world conditions of modulated co-channel interference and low-grade hardware. However, this only holds if the training dataset fully captures the variety of interference and hardware types in the real radio environment; otherwise, the DMC performance deteriorates significantly. Our work has two important engineering implications. First, it shows that it is not straightforward to exchange learned classifier models among dissimilar radio environments and devices in practice. Second, our analysis suggests that the key missing link for real-world deployment of DMC is designing signal features that generalize well to diverse wireless network scenarios. We are making our RF signal dataset publicly available as a step towards a unified framework for realistic DMC evaluation.