Incremental Machine Speech Chain Towards Enabling Listening while Speaking in Real-time

TL;DR

This paper introduces an incremental machine speech chain framework that enables real-time listening and speaking, reducing delays caused by long utterances while maintaining performance.

Contribution

It proposes incremental ASR and TTS systems that improve together in real-time, addressing delay issues in previous non-incremental frameworks.

Findings

Reduces delay in processing long utterances

Maintains comparable performance to non-incremental systems

Enables real-time listening and speaking in machine speech systems

Abstract

Inspired by a human speech chain mechanism, a machine speech chain framework based on deep learning was recently proposed for the semi-supervised development of automatic speech recognition (ASR) and text-to-speech synthesis TTS) systems. However, the mechanism to listen while speaking can be done only after receiving entire input sequences. Thus, there is a significant delay when encountering long utterances. By contrast, humans can listen to what hey speak in real-time, and if there is a delay in hearing, they won't be able to continue speaking. In this work, we propose an incremental machine speech chain towards enabling machine to listen while speaking in real-time. Specifically, we construct incremental ASR (ISR) and incremental TTS (ITTS) by letting both systems improve together through a short-term loop. Our experimental results reveal that our proposed framework is able to reduce delays due to long utterances while keeping a comparable performance to the non-incremental basic machine speech chain.