mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR

TL;DR

This paper introduces RAD-GAN, a novel two-stage GAN-based method tailored for reconstructing high-quality speech from low SNR mmWave radar signals, outperforming existing approaches.

Contribution

The paper presents a new mmWave-specific GAN architecture with a Multi-Mel Discriminator and Residual Fusion Gate, enabling effective bandwidth extension and noise reduction in speech signals.

Findings

RAD-GAN outperforms state-of-the-art methods in low SNR conditions

The approach works without pre-trained modules or data augmentation

Effective on signals captured through glass walls

Abstract

Millimeter-wave (mmWave) radar captures are band-limited and noisy, making for difficult reconstruction of intelligible full-bandwidth speech. In this work, we propose a two-stage speech reconstruction pipeline for mmWave using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios (-5 dB to -1 dB), captured through glass walls. We propose an mmWave-tailored Multi-Mel Discriminator (MMD) and a Residual Fusion Gate (RFG) to enhance the generator input to process multiple conditioning channels. The proposed two-stage pipeline involves pretraining the model on synthetically clipped clean speech and finetuning on fused mel spectrograms generated by the RFG. We empirically show that the proposed method, trained on a limited dataset, with no pre-trained modules, and no data augmentations, outperformed state-of-the-art approaches for this specific task. Audio examples of RAD-GAN are available online at https://rad-gan-demo-site.vercel.app/.