Exploring the Benefits of Tokenization of Discrete Acoustic Units

TL;DR

This paper demonstrates that tokenizing discrete acoustic units enhances performance and efficiency in speech-related tasks, supported by empirical results and theoretical insights.

Contribution

It introduces the application of tokenization to phonetic units and DAUs, showing significant improvements in speech prediction tasks.

Findings

Tokenization improves prediction accuracy.

Tokenization speeds up training and inference.

Theoretical analysis explains performance gains.

Abstract

Tokenization algorithms that merge the units of a base vocabulary into larger, variable-rate units have become standard in natural language processing tasks. This idea, however, has been mostly overlooked when the vocabulary consists of phonemes or Discrete Acoustic Units (DAUs), an audio-based representation that is playing an increasingly important role due to the success of discrete language-modeling techniques. In this paper, we showcase the advantages of tokenization of phonetic units and of DAUs on three prediction tasks: grapheme-to-phoneme, grapheme-to-DAUs, and unsupervised speech generation using DAU language modeling. We demonstrate that tokenization yields significant improvements in terms of performance, as well as training and inference speed, across all three tasks. We also offer theoretical insights to provide some explanation for the superior performance observed.