Comparing Unsupervised and Supervised Semantic Speech Tokens: A Case Study of Child ASR

TL;DR

This study compares supervised and unsupervised semantic speech tokens for child ASR, revealing supervised methods outperform unsupervised ones and even surpass continuous representations, especially in low-resource and ultra-low bitrate scenarios.

Contribution

It provides a systematic comparison of supervised and unsupervised semantic speech tokens, demonstrating the superiority of supervised methods for child ASR tasks.

Findings

Supervised semantic tokens outperform unsupervised tokens in child ASR.

Supervised methods surpass continuous speech representations.

Supervised tokens perform well even at ultra-low bitrates.

Abstract

Discrete speech tokens have gained attention for their storage efficiency and integration with Large Language Models (LLMs). They are commonly categorized into acoustic and semantic tokens, with the latter being more advantageous for Automatic Speech Recognition (ASR). Traditionally, unsupervised K-means clustering has been used to extract semantic speech tokens from Speech Foundation Models (SFMs). Recently, supervised methods, such as finite scalar quantization (FSQ) trained with ASR loss, have emerged for speech generation. Both approaches leverage pre-trained SFMs, benefiting low-resource tasks such as child ASR. This paper systematically compares supervised and unsupervised semantic speech tokens for child ASR. Results show that supervised methods not only outperform unsupervised ones but even unexpectedly surpass continuous representations, and they perform well even in ultra-low bitrate settings. These findings highlight the advantages of supervised semantic tokens and offer insights for improving discrete speech tokenization.