SPAM: Style Prompt Adherence Metric for Prompt-based TTS

TL;DR

This paper introduces SPAM, an automatic metric for evaluating how well prompt-based TTS models adhere to style cues, ensuring both plausibility and faithfulness aligned with human judgment.

Contribution

The paper proposes SPAM, a novel metric inspired by CLAP, that explicitly measures style prompt adherence in TTS, addressing limitations of prior evaluation methods.

Findings

SPAM correlates strongly with human MOS scores.

SPAM effectively discriminates different style semantics.

SPAM provides a grounded, automatic evaluation for TTS style adherence.

Abstract

Prompt-based text-to-speech (TTS) aims to generate speech that adheres to fine-grained style cues provided in a text prompt. However, most prior works depend on neither plausible nor faithful measures to evaluate prompt adherence. That is, they cannot ensure whether the evaluation is grounded on the prompt and is similar to a human. Thus, we present a new automatic metric, the Style Prompt Adherence Metric, which explicitly satisfies both plausibility and faithfulness. Inspired by the CLAP, our approach factorizes speech into acoustic attributes and aligns them with the style prompt. Also, we trained the scorer with a supervised contrastive loss, which could provide a clearer distinction between different semantics. We conducted two experiments on two perspectives. The plausibility experiment showed that SPAM achieved a strong correlation with the mean opinion score (MOS). Also, the faithfulness experiment demonstrated that SPAM is successfully grounded to the given style prompt, as it can discriminate different semantics of the prompt. We believe that SPAM can provide a viable automatic solution for evaluating style prompt adherence of synthesized speech.