Are These Even Words? Quantifying the Gibberishness of Generative Speech Models

TL;DR

This paper investigates how to quantify gibberishness in generative speech models using unsupervised methods and language models, addressing the challenge of detecting implausible or nonsensical speech artifacts.

Contribution

It introduces a novel unsupervised approach leveraging language models to measure gibberishness in speech, and provides a new dataset of high-quality synthesized gibberish speech.

Findings

Proposes a method to quantify gibberishness without supervision.

Publishes a dataset of synthesized gibberish speech.

Provides code for scoring speech using language models.

Abstract

Significant research efforts are currently being dedicated to non-intrusive quality and intelligibility assessment, especially given how it enables curation of large scale datasets of in-the-wild speech data. However, with the increasing capabilities of generative models to synthesize high quality speech, new types of artifacts become relevant, such as generative hallucinations. While intrusive metrics are able to spot such sort of discrepancies from a reference signal, it is not clear how current non-intrusive methods react to high-quality phoneme confusions or, more extremely, gibberish speech. In this paper we explore how to factor in this aspect under a fully unsupervised setting by leveraging language models. Additionally, we publish a dataset of high-quality synthesized gibberish speech for further development of measures to assess implausible sentences in spoken language, alongside code for calculating scores from a variety of speech language models.