Streaming Parrotron for on-device speech-to-speech conversion

TL;DR

This paper introduces a streaming speech-to-speech conversion model that operates on mobile devices with minimal delay and high efficiency, enabling real-time on-device speech normalization and synthesis.

Contribution

The paper proposes a novel streaming-based approach with a hybrid look-ahead mechanism for real-time speech conversion on mobile devices, achieving near real-time performance.

Findings

Almost 2x faster than real time on Pixel4 CPU

Minimal loss in speech quality compared to non-streaming models

Effective hybrid look-ahead approach for low-latency processing

Abstract

We present a fully on-device streaming Speech2Speech conversion model that normalizes a given input speech directly to synthesized output speech. Deploying such a model on mobile devices pose significant challenges in terms of memory footprint and computation requirements. We present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a reference state of the art non-streaming approach. Our method consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech. To achieve an acceptable delay-quality trade-off, we propose a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We show that our streaming approach is almost 2x faster than real time on the Pixel4 CPU.