Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals

TL;DR

Radio2Text introduces a novel mmWave-based streaming speech recognition system capable of handling large vocabularies with low latency, utilizing a tailored Transformer architecture and cross-modal knowledge distillation to improve accuracy.

Contribution

The paper presents the first mmWave-based streaming ASR system with a large vocabulary, employing a new guidance initialization and cross-modal knowledge distillation techniques.

Findings

Achieves 5.7% character error rate on large vocabulary recognition.

Demonstrates effective learning of speech features from mmWave signals.

Enhances streaming ASR performance with novel transfer learning methods.

Abstract

Millimeter wave (mmWave) based speech recognition provides more possibility for audio-related applications, such as conference speech transcription and eavesdropping. However, considering the practicality in real scenarios, latency and recognizable vocabulary size are two critical factors that cannot be overlooked. In this paper, we propose Radio2Text, the first mmWave-based system for streaming automatic speech recognition (ASR) with a vocabulary size exceeding 13,000 words. Radio2Text is based on a tailored streaming Transformer that is capable of effectively learning representations of speech-related features, paving the way for streaming ASR with a large vocabulary. To alleviate the deficiency of streaming networks unable to access entire future inputs, we propose the Guidance Initialization that facilitates the transfer of feature knowledge related to the global context from the non-streaming Transformer to the tailored streaming Transformer through weight inheritance. Further, we propose a cross-modal structure based on knowledge distillation (KD), named cross-modal KD, to mitigate the negative effect of low quality mmWave signals on recognition performance. In the cross-modal KD, the audio streaming Transformer provides feature and response guidance that inherit fruitful and accurate speech information to supervise the training of the tailored radio streaming Transformer. The experimental results show that our Radio2Text can achieve a character error rate of 5.7% and a word error rate of 9.4% for the recognition of a vocabulary consisting of over 13,000 words.