mmWave-Whisper: Phone Call Eavesdropping and Transcription Using Millimeter-Wave Radar

TL;DR

mmWave-Whisper demonstrates a novel radar-based system capable of eavesdropping on phone calls and transcribing speech remotely, achieving full-vocabulary recognition using millimeter-wave radar and advanced speech models.

Contribution

This work is the first to enable full-sentence speech recognition from smartphone earpiece vibrations using millimeter-wave radar and synthetic training data techniques.

Findings

Achieves 44.74% word accuracy in real-world scenarios

Operates effectively within 25-125 cm range

Utilizes synthetic data and domain adaptation for training

Abstract

This paper introduces mmWave-Whisper, a system that demonstrates the feasibility of full-corpus automated speech recognition (ASR) on phone calls eavesdropped remotely using off-the-shelf frequency modulated continuous wave (FMCW) millimeter-wave radars. Operating in the 77-81 GHz range, mmWave-Whisper captures earpiece vibrations from smartphones, converts them into audio, and processes the audio to produce speech transcriptions automatically. Unlike previous work that focused on loudspeakers or limited vocabulary, this is the first work to perform such a speech recognition by handling large vocabulary and full sentences on earpiece vibrations from smartphones. This approach expands the potential of radar-audio eavesdropping. mmWave-Whisper addresses challenges such as the lack of large scale training datasets, low SNR, and limited frequency information in radar data through a systematic pipeline designed to leverage synthetic training data, domain adaptation, and inference by incorporating OpenAI's Whisper automatic speech recognition model. The system achieves a word accuracy rate of 44.74% and a character accuracy rate of 62.52% over a range of 25 cm to 125 cm. The paper highlights emerging misuse modalities of AI as the technology evolves rapidly.