Morphologically-Informed Tokenizers for Languages with Non-Concatenative Morphology: A case study of Yolox\'ochtil Mixtec ASR

TL;DR

This paper introduces two novel, nonlinear, morphologically-informed tokenizers for Yoloxóchitl Mixtec ASR, demonstrating their competitiveness with traditional models and analyzing their impact on ASR performance and linguistic information retention.

Contribution

The paper presents two innovative nonlinear tokenization schemes tailored for non-concatenative morphology, improving ASR performance and reducing annotation workload.

Findings

Segment-and-Melody tokenizer outperforms traditional tokenizers in word error rate.

Tokenizers show competitive performance with BPE and Unigram models.

Morphologically-informed tokenizers correlate with downstream ASR performance.

Abstract

This paper investigates the impact of using morphologically-informed tokenizers to aid and streamline the interlinear gloss annotation of an audio corpus of Yolox\'ochitl Mixtec (YM) using a combination of ASR and text-based sequence-to-sequence tools, with the goal of improving efficiency while reducing the workload of a human annotator. We present two novel tokenization schemes that separate words in a nonlinear manner, preserving information about tonal morphology as much as possible. One of these approaches, a Segment and Melody tokenizer, simply extracts the tones without predicting segmentation. The other, a Sequence of Processes tokenizer, predicts segmentation for the words, which could allow an end-to-end ASR system to produce segmented and unsegmented transcriptions in a single pass. We find that these novel tokenizers are competitive with BPE and Unigram models, and the Segment-and-Melody model outperforms traditional tokenizers in terms of word error rate but does not reach the same character error rate. In addition, we analyze tokenizers on morphological and information-theoretic metrics to find predictive correlations with downstream performance. Our results suggest that nonlinear tokenizers designed specifically for the non-concatenative morphology of a language are competitive with conventional BPE and Unigram models for ASR. Further research will be necessary to determine the applicability of these tokenizers in downstream processing tasks.