Bayesian Speech Synthesizers Can Learn from Multiple Teachers

TL;DR

This paper introduces BELLE, a Bayesian TTS framework that models speech uncertainty, improves naturalness, and outperforms larger models by leveraging a novel training strategy and probabilistic modeling.

Contribution

BELLE is the first Bayesian TTS model that captures data-dependent uncertainty without increasing inference latency or model size.

Findings

BELLE reduces WER by 25.8% compared to larger models.

It effectively models speech variability using a Normal-Inverse-Gamma distribution.

Supports high-quality streaming speech synthesis.

Abstract

Text-to-Speech (TTS) is inherently a "one-to-many" mapping characterized by intrinsic uncertainty, yet current paradigms often oversimplify it into a deterministic regression task. While continuous-valued autoregressive (AR) models have recently emerged as a promising alternative to discrete codec-based approaches, they typically rely on a fixed-variance prior, fundamentally constraining generation to a static point estimate that ignores the dynamic variability of natural speech. To bridge this gap, we propose BELLE (Bayesian evidential learning with language modelling), a framework that shifts from deterministic prediction to principled Bayesian inference without increasing model parameters or inference latency. By modeling the acoustic target as a Normal-Inverse-Gamma distribution, BELLE captures data-dependent aleatoric uncertainty. To enable accurate variance estimation on standard single-reference datasets, we introduce a "one-to-many" training strategy that leverages synthetic samples as a statistical support set, allowing the model to learn robust distributional properties rather than merely imitating teacher artifacts. Experiments demonstrate that BELLE, trained on only ~5k hours of data, outperforms leading open-source models trained on 50k hours (achieving a 25.8% relative WER reduction) and naturally supports high-quality streaming generation. Audio samples are available at https://belletts.github.io/Belle/.