Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow

TL;DR

VarianceFlow is a novel text-to-speech model that combines normalizing flow with variance control, leading to more precise speech synthesis and better controllability compared to existing methods.

Contribution

It introduces VarianceFlow, integrating normalizing flow with variance modeling to enhance speech quality and control in TTS systems.

Findings

Outperforms state-of-the-art TTS models in speech quality

Provides more accurate variance control for speech attributes

Achieves superior controllability and naturalness in synthesized speech

Abstract

There are two types of methods for non-autoregressive text-to-speech models to learn the one-to-many relationship between text and speech effectively. The first one is to use an advanced generative framework such as normalizing flow (NF). The second one is to use variance information such as pitch or energy together when generating speech. For the second type, it is also possible to control the variance factors by adjusting the variance values provided to a model. In this paper, we propose a novel model called VarianceFlow combining the advantages of the two types. By modeling the variance with NF, VarianceFlow predicts the variance information more precisely with improved speech quality. Also, the objective function of NF makes the model use the variance information and the text in a disentangled manner resulting in more precise variance control. In experiments, VarianceFlow shows superior performance over other state-of-the-art TTS models both in terms of speech quality and controllability.

Tables

Table 1. Table 1 : 9-scale MOS results with 95% confidence intervals.

Model	MOS
GT Waveform	4.47 $\pm$ 0.07
GT Melspectrogram	4.34 $\pm$ 0.08
Tacotron 2	4.03 $\pm$ 0.07
Glow-TTS	3.72 $\pm$ 0.13
FastSpeech 2-phoneme	3.92 $\pm$ 0.07
FastSpeech 2-frame	3.66 $\pm$ 0.09
VarianceFlow-phoneme	4.04 $\pm$ 0.08
VarianceFlow-frame	4.19 $\pm$ 0.07

Table 2. Table 2 : FFE (%) and 9-scale MOS results measured with different pitch shift scale λ 𝜆 \lambda .

Model	$𝝀=-\mathrm{𝟔}$		$𝝀=-\mathrm{𝟒}$		$𝝀=-\mathrm{𝟐}$		$𝝀=+\mathrm{𝟐}$		$𝝀=+\mathrm{𝟒}$		$𝝀=+\mathrm{𝟔}$
	FFE	MOS	FFE	MOS	FFE	MOS	FFE	MOS	FFE	MOS	FFE	MOS
FastSpeech 2	15.94	2.93	14.00	3.46	12.61	3.65	10.94	3.29	11.57	2.63	12.24	2.04
VarianceFlow-reversed	21.93	3.77	35.97	4.01	53.47	4.00	66.37	3.90	67.07	3.69	67.03	3.53
VarianceFlow	24.14	3.43	12.16	3.87	9.02	4.05	7.26	3.95	7.52	3.39	7.68	2.74

Equations

$$\log {\mathbf{p}}_{\mathbf{\theta }}(\mathbf{x}|\mathbf{h})=\log \mathbf{p}(\mathbf{z})+\sum _{i=1}^k\log |\det (\mathbf{J}({\mathbf{f}}_{\mathbf{i}}(\mathbf{x};\mathbf{h})))|$$

$$\mathbf{z}={\mathbf{f}}_{\mathbf{k}}\circ {\mathbf{f}}_{\mathbf{k}-1}\circ \dots \circ {\mathbf{f}}_0(\mathbf{x};\mathbf{h}),$$

$${\mathcal{L}}_{total}={\mathcal{L}}_{melspec}+{\mathcal{L}}_{duration}+\alpha \cdot {\mathcal{L}}_{pitch}+\alpha \cdot {\mathcal{L}}_{energy},$$

$${\mathcal{L}}_{NF}=D_{KL}\left[{\mathbf{q}}_{\mathbf{\theta }}(\mathbf{z}|\mathbf{h})\parallel \mathbf{p}(\mathbf{z})\right]+H(\mathbf{x}|\mathbf{h})$$

$$f_{\lambda }=2^{\frac{\lambda }{12}}\times f_0$$

Full text

**Index Terms— ** Text-to-Speech, Speech Synthesis, Normalizing Flow, Generative Model

1 Introduction

In Text-to-Speech (TTS), the one-to-many relationship between text and speech is one of the major problems that makes it challenging to learn the text-to-speech conversion. The early autoregressive (AR) TTS models [1, 2] dealt with the difficulty by factorizing the speech distribution into the product of homogeneous conditional factors in sequential order. However, although they succeeded in generating high-quality speech, their slow inference speed and exposure bias were inevitable problems inherent in the AR models.

As it has advanced from AR TTS models to non-AR TTS models, two types of methods for the non-AR models to solve the one-to-many problem have been proposed. The first type (Type-I) is to use an advanced generative framework such as normalizing flow [3], diffusion model [4], and generative adversarial network [5]. Unlike the mean squared error (MSE) based training that assumes a Gaussian distribution, these frameworks do not assume any pre-defined distribution for a target distribution. As a result, they can generate high-quality and diverse speech samples compared to the previous non-AR TTS models trained with the MSE loss [6, 7]. The second type of the methods (Type-II) is to solve a task of TTS by dividing it into simpler two tasks based on variance information such as pitch or energy [8, 9, 10]: (1) text conditioned variance modeling; (2) text and variance information conditioned speech generation. Then, by using ground-truth variance information when learning the two tasks, Type-II models achieve faster training convergence and higher speech quality. Moreover, unlike Type-I models, Type-II models can explicitly control the variance factors by manipulating the variance values used in the speech generation.

In this paper, we propose VarianceFlow, a novel model that combines the advantages of Type-I and Type-II, achieving a high-quality and controllable TTS model. Unlike the previous Type-II models that learn the text conditioned variance modeling based on the MSE loss, our model uses normalizing flow (NF) for the variance modeling (Figure 1).

There are two large advantages of using NF for the variance modeling. First, since NF is robust to the one-to-many problem, it learns the variance distribution better than using the MSE loss [11], and it leads to improved speech quality. Second, NF enhances the variance controllability by disentangling the latent variance representation and the text [12].

In experiments, VarianceFlow shows its superiority in variance modeling by showing better speech quality compared to other AR and non-AR state-of-the-art TTS models. In addition, VarianceFlow shows its more precise variance controllability by showing that it uses the provided pitch values more intactly. Lastly, we show that VarianceFlow can generate diverse speech samples given a text input.

2 VarianceFlow

In this section, we first explain FastSpeech 2 [8], which is a baseline of our model. Then, we present VarianceFlow that learns to match a variance distribution to a prior distribution based on normalizing flow.

2.1 FastSpeech 2

FastSpeech 2 is a Type-II non-AR TTS model that uses pitch and energy as additional variance information in speech generation111Phoneme durations are also used as variance information, but it is out of focus of this work.. Firstly, a phoneme sequence is encoded by an FFT encoder, and it is expanded to the length of the target melspectrogram given phoneme durations. Next, ground-truth pitch and energy values extracted in frame-level are projected to the space of the text representation. Lastly, the projected variance vectors are added to the expanded text representation, and an FFT decoder generates a melspectrogram based on it.

Since the ground-truth variance information is unavailable at inference, FastSpeech 2 has additional modules called variance predictor for variance modeling. During training, the variance predictors are trained with the MSE loss to predict the ground-truth variance values based on the input text. Then, the variance values predicted by the variance predictors are used at inference. However, there is still a problem that predicting pitch or energy from a text is also a difficult task having a one-to-many relationship. For this, FastSpeech 2 trains its pitch predictor with wavelet transformed pitch spectrogram instead of raw pitch values. Also, FastPitch [9], which uses pitch information similar to FastSpeech 2, uses phoneme-averaged pitch values to ease the difficulty of learning the complex pitch distribution.

2.2 VarianceFlow

VarianceFlow is a model that uses variance information such as pitch and energy based on normalizing flow [13, 11]. During training, pitch and energy variance information is provided to a feed-forward transformer (FFT) decoder via NF modules (Figure 1). Then, VarianceFlow learns to generate speech based on the information and the input text. Meanwhile, the NF modules learn to match a latent variance distribution to a simple prior distribution. At inference, the variance information is provided to the FFT blocks by directly sampling the latent representation from the prior.

A normalizing flow (NF) consisting of consecutive bijective transforms converts a complex variance distribution to a simple prior distribution. Based on the change of variables formula, using bijective transforms enables to calculate probability density of the variance information by setting the latent variance distribution as a simple prior distribution:

[TABLE]

[TABLE]

where $\boldsymbol{x}$ is a variance factor, $\boldsymbol{h}$ is a hidden representation, z is a latent representation, and $\boldsymbol{f_{i}}$ is a bijective transform. Therefore, using a unit Gaussian distribution as the prior distribution, VarianceFlow is trained to maximize the log-likelihood of the variance information based on (1) and (2). To fully utilize parallel computation with enough expressive power, VarianceFlow uses rational-quadratic coupling transform [14].

2.3 Loss function

Replacing the MSE losses for pitch and energy predictors in FastSpeech 2 [8] with the NF losses of VarianceFlow, the final objective function of VarianceFlow becomes as follows:

[TABLE]

where the first two terms are the losses existing in FastSpeech 2, and the last two terms are the negative log-likelihood NF losses for pitch and energy. Here, each of the NF losses ($\mathcal{L}_{NF}$) not only learns the variance modeling but also disentangles the variance factor with the text, and it is shown by the fact that $\mathcal{L}_{NF}$ can be decomposed as follows [12]:

[TABLE]

For an NF module in VarianceFlow, the second entropy term is constant, so it is trained to minimize the kullback-leibler divergence between the conditional latent variance distribution $\boldsymbol{q_{\theta}(z|h)}$ and the prior distribution $\boldsymbol{p(z)}$. It means that it is trained to make $\boldsymbol{z}$ and $\boldsymbol{h}$ disentangled because the prior $\boldsymbol{p(z)}$ is chosen independently with $\boldsymbol{h}$. As a result, it enhances the responsiveness of the model to $\boldsymbol{z}$, resulting in more precise control.

2.4 Controlling a variance factor

The invertibility of NF makes it possible to obtain the raw variance value $\boldsymbol{x}$ from the latent representation $\boldsymbol{z}$. Therefore, it is also possible for VarianceFlow to control the variance factors as other Type-II TTS models (Figure 2). To control the variance information, the sampled latent representations are first brought to the raw variance space using the inverse transforms of NF. Then, the variance factor is manipulated in the space (e.g., multiplying a constant to the raw variance value $\boldsymbol{x}$). Lastly, the manipulated value is provided to the FFT decoder via NF again.

3 Experiments

In this section, experimental setup and results are provided. Also, the audio samples and more experimental figures are available at https://leeyoonhyung.github.io/VarianceFlow-demo.

3.1 Experimental setup

In experiments, LJSpeech dataset [15] is used. The waves are converted to log-melspectrograms with 1024 fft window and 256 hop lengths. The pitch values are extracted using Parselmouth [16] and they are converted to log-scale and pitch spectrogram is not used. When calculating the total loss of VarianceFlow, $\alpha=0.1$ is used for the coefficient of the NF losses. When generating audio samples, we use a zero-mean Gaussian prior distribution with a standard deviation $\sigma=0.333$, and we use HiFi-GAN vocoder [17]. As a baseline, we adopt FastSpecch 2 architecture [8] and also adopt its FFT architecture for VarianceFlow. For an NF module of VarianceFlow, we use 4-layer rational-quadratic coupling transform architecture following the architecture of the stochastic duration predictor in [18]. We train all models for 230k steps with a batch size of 16. We use AdamW optimizer [19] with $\beta_{1}=0.9$ and $\beta_{2}=0.98$ with the Noam learning rate scheduling [20].

3.2 Speech quality

To see the effectiveness of VarianceFlow in improving speech quality, we compare the speech quality of VarainceFlow with FastSpeech 2 based on the 9-scale mean opinion score (MOS), while varying the ways of using the variance information either phoneme-averaged level or frame level (Table 1). For each model, fifty samples from the test set are used, and five testers living in the U.S. are asked to give a score ranging from 1 to 5 to each audio sample. For FastSpeech 2, MOS is better when variance information is used in the phoneme-averaged level. It means that training the current architecture of the variance predictors using the MSE loss lacks in learning the complex variance distribution. On the contrary, VarianceFlow shows better speech quality when the variance information is provided in the frame level. It means that our VarianceFlow learns the complex variance distribution well based on NF. Although the architectures used in variance modeling are different, directly using the latent representation sampled from a Gaussian distribution does not slow down the synthesis speed or increase the number of parameters at inference. Overall, VarianceFlow trained with the frame level variance information shows the best speech quality including other state-of-the-art AR and non-AR TTS models, Tacotron 2 and Glow-TTS.

3.3 Controllability

To compare the controllability of the models, we conduct objective and subjective evaluations while adjusting the predicted pitch values with various pitch shift coefficients $\lambda$. In addition, to see the effectiveness of the NF loss in disentangling the variance factors with other factors, we also train a model by reverting the direction of NF (‘VarianceFlow-reversed’). For this model, the raw variance information is directly provided to the FFT decoder while the NF modules separately learn to match the latent variance distribution and the prior distribution. In this case, it still has the advantage of using advanced distribution modeling method, but it uses the variance information without disentangling.

3.3.1 Pitch responsiveness

To evaluate the responsiveness of the models to the variance information, we measure f0 frame error rate (FFE) [21] between the pitch values provided to the decoder and the pitch values extracted from the generated audio samples. All audio samples are generated by adjusting the pitch values in a semitone unit as follows [10]:

[TABLE]

where $f_{0}$ is the pitch value before shifted. In the experiments, $\lambda=\{-6,-4,-2,+2,+4,+6\}$ are used. As shown in Table 2 and Figure 3, VarianceFlow uses the provided pitch most intactly for most $\lambda$. Therefore, it is possible to control the pitch with VarianceFlow precisely. VarianceFlow-reversed shows the largest FFE, and it can be understood based on Figure 3. Figure 3 shows that VarianceFlow-reversed learns to use the variation of provided pitch, but it ignores the average of the provided pitch when generating a speech.

3.3.2 Speech quality with pitch shift

To evaluate the models subjectively, we measure the MOS for the samples generated using the shifted pitch values in the same way with Section 3.2. For each model and $\lambda$, twenty samples from the test set are used. The results in Table 2 show that our VarianceFlow achieves better speech quality for all $\lambda$ compared to FastSpeech 2, and even outperforms VarianceFlow-reversed for $\lambda=\{-2,+2\}$. We conjecture that this is because while the latent representation is passed to the FFT decoder, NF loss works as a regularization. Although VarianceFlow becomes worse than VarianceFlow-reversed when $\lambda$ gets larger, we attribute this to the fact that VarianceFlow-reversed does not follow the pitch shift.

3.4 Diversity

To see how the variation in latent space affects the variance factors, we generate diverse speech samples from a single text while using different standard deviation $\sigma=\{0.0,0.667\}$ for latent sampling (Figure 4). Here, the $\sigma$ values are chosen in a range where the speech quality is maintained. In the experiment, it is shown that VarianceFlow can generate natural speech without using latent representations, and pitch variance in samples appears when using the latent representations. However, the different samples have similar prosody and perceptually sound similar. We conjecture that this is because it is trained to minimize the conditional entropy $H(\boldsymbol{x}|\boldsymbol{h})$ (Eq. (4)). Therefore, we repeat the samplings while having the dropout layers in the FFT encoder activated [18, 22]. In this case, the diversity in terms of prosody increases with varying durations without much loss of speech quality.

4 Conclusion

We propose VarianceFlow, a novel model that uses variance information based on normalizing flow (NF). VarianceFlow shows that it learns the variance distribution better with improved speech quality. Also, the invertibility of NF maintains its ability to control the variance factors, and we show that the objective of NF even enhances the variance controllability.

5 Acknowledgments

K. Jung is with ASRI, Seoul National University, Korea. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (No. 2021R1A2C2008855). This research resulted from a study on the “HPC Support” Project, supported by the ‘Ministry of Science and ICT’ and NIPA.