Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation

TL;DR

This paper introduces a novel method for annotating Japanese speech data by fine-tuning a pre-trained ASR model with dictionary-enhanced decoding, improving label accuracy and TTS naturalness.

Contribution

It presents a new approach combining transcript-conditioned ASR fine-tuning with dictionary-based decoding for improved phonemic and prosodic annotation in Japanese speech datasets.

Findings

Outperforms previous text- or audio-only annotation methods.

Achieves comparable speech naturalness to manual annotations in TTS.

Demonstrates improved annotation accuracy through dictionary-augmented decoding.

Abstract

In this paper, we propose a method for annotating phonemic and prosodic labels on a given audio-transcript pair, aimed at constructing Japanese text-to-speech (TTS) datasets. Our approach involves fine-tuning a large-scale pre-trained automatic speech recognition (ASR) model, conditioned on ground truth transcripts, to simultaneously output phrase-level graphemes and annotation labels. To further correct errors in phonemic labeling, we employ a decoding strategy that utilizes dictionary prior knowledge. The objective evaluation results demonstrate that our proposed method outperforms previous approaches relying solely on text or audio. The subjective evaluation results indicate that the naturalness of speech synthesized by the TTS model, trained with labels annotated using our method, is comparable to that of a model trained with manual annotations.