CC-G2PnP: Streaming Grapheme-to-Phoneme and prosody with Conformer-CTC for unsegmented languages

TL;DR

CC-G2PnP is a streaming model that connects language models and text-to-speech for unsegmented languages, using Conformer-CTC to predict phonemes and prosody efficiently in real-time.

Contribution

It introduces a novel streaming G2PnP model based on Conformer-CTC that handles unsegmented languages without relying on explicit word boundaries.

Findings

Outperforms baseline in phoneme and prosody prediction accuracy

Effective for unsegmented languages like Japanese

Enables stable streaming inference with minimal look-ahead

Abstract

We propose CC-G2PnP, a streaming grapheme-to-phoneme and prosody (G2PnP) model to connect large language model and text-to-speech in a streaming manner. CC-G2PnP is based on Conformer-CTC architecture. Specifically, the input grapheme tokens are processed chunk by chunk, which enables streaming inference of phonemic and prosodic (PnP) labels. By guaranteeing minimal look-ahead size to each input token, the proposed model can consider future context in each token, which leads to stable PnP label prediction. Unlike previous streaming methods that depend on explicit word boundaries, the CTC decoder in CC-G2PnP effectively learns the alignment between graphemes and phonemes during training, making it applicable to unsegmented languages. Experiments on a Japanese dataset, which has no explicit word boundaries, show that CC-G2PnP significantly outperforms the baseline streaming G2PnP model in the accuracy of PnP label prediction.