Self-Supervised Radio Pre-training: Toward Foundational Models for Spectrogram Learning

TL;DR

This paper introduces a self-supervised pretraining method for radio spectrogram models using Masked Spectrogram Modeling, enabling improved performance in spectrum forecasting and segmentation tasks.

Contribution

It presents a novel self-supervised learning approach for radio spectrograms with a Convolutional LSTM architecture, advancing foundational models in radio signal processing.

Findings

Achieves competitive spectrum forecasting accuracy

Demonstrates effective radio signal segmentation

Validates the approach's potential for foundational radio models

Abstract

Foundational deep learning (DL) models are general models, trained on large, diverse, and unlabelled datasets, typically using self-supervised learning techniques have led to significant advancements especially in natural language processing. These pretrained models can be fine-tuned for related downstream tasks, offering faster development and reduced training costs, while often achieving improved performance. In this work, we introduce Masked Spectrogram Modeling, a novel self-supervised learning approach for pretraining foundational DL models on radio signals. Adopting a Convolutional LSTM architecture for efficient spatio-temporal processing, we pretrain the model with an unlabelled radio dataset collected from over-the-air measurements. Subsequently, the pretrained model is fine-tuned for two downstream tasks: spectrum forecasting and segmentation. Experimental results demonstrate that our methodology achieves competitive performance in both forecasting accuracy and segmentation, validating its effectiveness for developing foundational radio models.