Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling

TL;DR

This paper explores combining adversarial training and disentangled speech representations to improve unsupervised subword discovery from speech, achieving significant improvements in speaker-invariant features and speech synthesis quality.

Contribution

It introduces a novel combination of adversarial training and FHVAE-based disentanglement within a unified framework for robust subword modeling.

Findings

Both approaches effectively improve speaker invariance.

Their combination yields marginal further improvements.

The methods outperform the official baseline in ABX discriminability.

Abstract

This study addresses the problem of unsupervised subword unit discovery from untranscribed speech. It forms the basis of the ultimate goal of ZeroSpeech 2019, building text-to-speech systems without text labels. In this work, unit discovery is formulated as a pipeline of phonetically discriminative feature learning and unit inference. One major difficulty in robust unsupervised feature learning is dealing with speaker variation. Here the robustness towards speaker variation is achieved by applying adversarial training and FHVAE based disentangled speech representation learning. A comparison of the two approaches as well as their combination is studied in a DNN-bottleneck feature (DNN-BNF) architecture. Experiments are conducted on ZeroSpeech 2019 and 2017. Experimental results on ZeroSpeech 2017 show that both approaches are effective while the latter is more prominent, and that their combination brings further marginal improvement in across-speaker condition. Results on ZeroSpeech 2019 show that in the ABX discriminability task, our approaches significantly outperform the official baseline, and are competitive to or even outperform the official topline. The proposed unit sequence smoothing algorithm improves synthesis quality, at a cost of slight decrease in ABX discriminability.

Tables

Table 1. Table 1: ZeroSpeech 2019 datasets

		English		Surprise
		Duration	#speakers	Duration	#speakers
Training	Unit	$15.5$ hrs	$100$	$15$ hrs	$112$
	Voice	$4.5$ hrs	$2$	$1.5$ hrs	$1$
Test		$0.5$ hr	$24$	$0.5$ hr	$15$

Table 2. Table 2: Comparison of baseline, topline and our submission

	English					Surprise
	ABX	Bitrate	MOS	CER	SS	ABX	Bitrate	MOS	CER	SS
Baseline [3]	$35.63$	$71.98$	$2.50$	$0.75$	$2.97$	$27.46$	$74.55$	$2.07$	$0.62$	$3.41$
Topline [3]	$29.85$	$37.73$	$2.77$	$0.44$	$2.99$	$16.09$	$35.20$	$3.92$	$0.28$	$3.95$
Unit	$21.00$	$413.2$	$1.56$	$0.84$	$2.18$	$10.64$	$470.2$	$1.28$	$0.74$	$2.01$
Unit+SM	$25.03$	$320.0$	$1.78$	$0.76$	$2.33$	$16.87$	$299.2$	$1.67$	$0.66$	$2.60$
PG	$13.82$	$1733$	$-$	$-$	$-$	$7.49$	$1373$	$-$	$-$	$-$
BNF	$13.33$	$1733$	$-$	$-$	$-$	$6.52$	$1373$	$-$	$-$	$-$

Equations

$$p_{\theta }({\mathbf{\mu }}_2)\prod _{n=1}^N{p}_{\theta }({\mathbf{z}}_1^{\mathbf{n}})p_{\theta }({\mathbf{z}}_2^{\mathbf{n}}|{\mathbf{\mu }}_2)p_{\theta }({\mathbf{x}}^{\mathbf{n}}|{\mathbf{z}}_1^{\mathbf{n}},{\mathbf{z}}_2^{\mathbf{n}}).$$

$$q_{\varphi }({\mathbf{\mu }}_2)\prod _{n=1}^N{q}_{\varphi }({\mathbf{z}}_2^{\mathbf{n}}|{\mathbf{x}}^{\mathbf{n}})q_{\varphi }({\mathbf{z}}_1^{\mathbf{n}}|{\mathbf{x}}^{\mathbf{n}},{\mathbf{z}}_2^{\mathbf{n}}).$$

$${\theta }_p$$

$${\theta }_h$$

Full text

Index Terms: acoustic unit discovery, subword modeling, zero resource, adversarial training, disentangled representation

1 Introduction

Nowadays speech processing is dominated by deep learning techniques. Deep neural network (DNN) acoustic models (AMs) for the tasks of automatic speech recognition (ASR) and speech synthesis have shown impressive performance for major languages such as English and Mandarin. Typically, training a DNN AM requires large amounts of transcribed data. For a large number of low-resource languages, for which very limited or no transcribed data are available, conventional methods of acoustic modeling are ineffective or even inapplicable.

In recent years, there has been an increasing research interest in zero-resource speech processing, i.e., only a limited amount of raw speech data (e.g. hours or tens of hours) are given while no text transcriptions or linguistic knowledge are available. The Zero Resource Speech Challenges (ZeroSpeech) 2015 [1], 2017 [2] and 2019 [3] precisely focus on this area. One problem tackled by ZeroSpeech 2015 and 2017 is subword modeling, learning frame-level speech representation that is discriminative to subword units and robust to linguistically-irrelevant factors such as speaker change. The latest challenge ZeroSpeech 2019 goes a step further by aiming at building text-to-speech (TTS) systems without any text labels (TTS without T) or linguistic expertise. Specifically, one is required to build an unsupervised subword modeling sub-system to automatically discover phoneme-like units in the concerned language, followed by applying the learned units altogether with speech data from which the units are inferred to train a TTS. Solving this problem may partially assist psycholinguists in understanding young children’s language acquisition mechanism [3].

This study addresses unsupervised subword modeling in ZeroSpeech 2019, which is also referred to as acoustic unit discovery (AUD). It is an essential problem and forms the basis of TTS without T. The exact goal of this problem is to represent untranscribed speech utterances by discrete subword unit sequences, which is slightly different from subword modeling in the contexts of ZeroSpeech 2017 & 2015. In practice, it can be formulated as an extension to the previous two challenges. For instance, after learning the subword discriminative feature representation at frame-level, the discrete unit sequences can be inferred by applying vector quantization methods followed by collapsing consecutive repetitive symbolic patterns. In the previous two challenges, several unsupervised representation learning approaches were proposed for comparison, such as cluster posteriorgrams (PGs) [4, 5, 6], DNN bottleneck features [7, 8], autoencoders (AEs) [9, 10], variational AEs (VAEs) [11, 12] and siamese networks [13, 14, 15].

One major difficulty in unsupervised subword modeling is dealing with speaker variation. The huge performance degradation caused by speaker variation reported in ZeroSpeech 2017 [2] implies that speaker-invariant representation learning is crucial and remains to be solved. In ZeroSpeech 2019, speaker-independent subword unit inventory is highly desirable in building a TTS without T system. In the literature, many works focused on improving the robustness of unsupervised feature learning towards speaker variation. One direction is to apply linear transform methods. Heck et al. [6] estimated fMLLR features in an unsupervised manner. Works in [7, 16] estimated fMLLR using a pre-trained out-of-domain ASR. Chen et al. [8] applied vocal tract length normalization (VTLN). Another direction is to employ DNNs. Zeghidour et al. [14] proposed to train subword and speaker same-different tasks within a triamese network and untangle linguistic and speaker information. Chorowski et al. [12] defined a speaker embedding as a condition of VAE decoder to free the encoder from capturing speaker information. Tsuchiya et al. [17] applied speaker adversarial training in a task related to the zero-resource scenario but transcription for a target language was used in model training.

In this paper, we propose to extend our recent research findings [11] on applying disentangled speech representation learned from factorized hierarchical VAE (FHVAE) models [18] to improve speaker-invariant subword modeling. The contributions made in this study are in several aspects. First, the FHVAE based speaker-invariant learning is compared with speaker adversarial training in the strictly unsupervised scenario. Second, the combination of adversarial training and disentangled representation learning is studied. Third, our proposed approaches are evaluated on the latest challenge ZeroSpeech 2019, as well as on ZeroSpeech 2017 for completeness. To our best knowledge, direct comparison of the two approaches and their combination has not been studied before.

2 System description

2.1 General framework

The general framework of our proposed approaches is illustrated in Figure 1. Given untranscribed speech data, the first step is to learn speaker-invariant features to support frame labeling. The FHVAE model [18] is adopted for this purpose. FHVAEs disentangle linguistic content and speaker information encoded in speech into different latent representations. Compared with raw MFCC features, FHVAE reconstructed features conditioned on latent linguistic representation are expected to keep linguistic content unchanged and are more speaker-invariant. Details of the FHVAE structure and feature reconstruction methods are described in Section 2.2.

The reconstructed features are fed as inputs to Dirichlet process Gaussian mixture model (DPGMM) [19] for frame clustering, as was done in [4]. The frame-level cluster labels are regarded as pseudo phone labels to support supervised DNN training. Motivated by successful applications of adversarial training [20] in a wide range of domain invariant learning tasks [21, 22, 23, 24], this work proposes to add an auxiliary adversarial speaker classification task to explicitly target speaker-invariant feature learning. After speaker adversarial multi-task learning (AMTL) DNN training, softmax PG representation from pseudo phone classification task is used to infer subword unit sequences. The resultant unit sequences are regarded as pseudo transcriptions for subsequent TTS training.

2.2 Speaker-invariant feature learning by FHVAEs

The FHVAE model formulates the generation process of sequential data by imposing sequence-dependent and sequence-independent priors to different latent variables [18]. It consists of an inference model $\phi$ and a generation model $\theta$. Let $\mathcal{D}=\{\bm{X^{i}}\}_{i=1}^{M}$ denote a speech dataset with $M$ sequences. Each $\bm{X^{i}}$ contains $N^{i}$ speech segments $\{\bm{x^{(i,n)}}\}^{N^{i}}_{n=1}$, where $\bm{x^{(i,n)}}$ is composed of fixed-length consecutive frames. The FHVAE model generates a sequence $\bm{X}$ from a random process as follows: (1) An s-vector $\bm{\mu_{2}}$ is drawn from a prior distribution $p_{\theta}(\bm{\mu_{2}})=\mathcal{N}(\bm{0},\sigma^{2}_{\mu_{2}}\bm{I})$; (2) Latent segment variables $\bm{z_{1}^{n}}$ and latent sequence variables $\bm{z_{2}^{n}}$ are drawn from $p_{\theta}(\bm{z_{1}^{n}})=\mathcal{N}(\bm{0},{\sigma^{2}_{z_{1}}}\bm{I})$ and $p_{\theta}(\bm{z_{2}^{n}|\bm{\mu_{2}}})=\mathcal{N}(\bm{\mu_{2}},{\sigma^{2}_{z_{2}}}\bm{I})$ respectively; (3) Speech segment $\bm{x^{n}}$ is drawn from $p_{\theta}(\bm{x^{n}}|\bm{z_{1}^{n},\bm{z_{2}^{n}}})=\mathcal{N}(f_{\mu_{x}}(\bm{z_{1}^{n}},\bm{z_{2}^{n}}),diag(f_{\sigma^{2}_{x}}(\bm{z_{1}^{n}},\bm{z_{2}^{n}}))$. Here $\mathcal{N}$ denotes standard normal distribution, $f_{\mu_{x}}(\cdot,\cdot)$ and $f_{\sigma^{2}_{x}}(\cdot,\cdot)$ are parameterized by DNNs. The joint probability for $\bm{X}$ is formulated as,

[TABLE]

Since the exact posterior inference is intractable, the FHVAE introduces an inference model $q_{\phi}$ to approximate the true posterior,

[TABLE]

Here $q_{\phi}(\bm{\mu_{2}}),q_{\phi}(\bm{z_{2}^{n}}|\bm{x^{n}})$ and $q_{\phi}(\bm{z_{1}^{n}}|\bm{x^{n}},\bm{z_{2}^{n}})$ are all diagonal Gaussian distributions. The mean and variance values of $q_{\phi}(\bm{z_{2}^{n}}|\bm{x^{n}})$ and $q_{\phi}(\bm{z_{1}^{n}}|\bm{x^{n}},\bm{z_{2}^{n}})$ are parameterized by two DNNs. For $q_{\phi}(\bm{\mu_{2}})$, during FHVAE training, a trainable lookup table containing posterior mean of $\bm{\mu_{2}}$ for each sequence is updated. During testing, maximum a posteriori (MAP) estimation is used to infer $\bm{\mu_{2}}$ for unseen test sequences. FHVAEs optimize the discriminative segmental variational lower bound which was defined in [18]. It contains a discriminative objective to prevent $\bm{z_{2}}$ from being the same for all utterances.

After FHVAE training, $\bm{z_{1}}$ encodes segment-level factors e.g. linguistic information, while $\bm{z_{2}}$ encodes sequence-level factors that are relatively consistent within an utterance. By concatenating training utterances of the same speaker into a single sequence for FHVAE training, the learned $\bm{\mu_{2}}$ is expected to be discriminative to speaker identity. This work considers applying s-vector unification [11] to generate reconstructed feature representation that keeps linguistic content unchanged and is more speaker-invariant than the original representation. Specifically, a representative speaker with his/her s-vector (denoted as $\bm{\mu_{2}^{*}}$) is chosen from the dataset. Next, for each speech segment $\bm{x^{(i,n)}}$ of an arbitrary speaker $i$, its corresponding latent sequence variable $\bm{z_{2}^{(i,n)}}$ inferred from $\bm{x^{(i,n)}}$ is transformed to $\bm{\hat{z}_{2}^{(i,n)}}=\bm{z_{2}^{(i,n)}}-\bm{\mu_{2}^{i}}+\bm{\mu_{2}^{*}}$, where $\bm{\mu_{2}^{i}}$ denotes the s-vector of speaker $i$. Finally the FHVAE decoder reconstructs speech segment $\bm{\hat{x}^{(i,n)}}$ conditioned on $\bm{z_{1}^{(i,n)}}$ and $\bm{\hat{z}_{2}^{(i,n)}}$. The features $\{\bm{\hat{x}^{(i,n)}}\}$ form our desired speaker-invariant representation.

2.3 Speaker AMTL

Speaker AMTL simultaneously trains a subword classification network ($M_{p}$), a speaker classification network ($M_{s}$) and a shared-hidden-layer feature extractor ($M_{h}$), where $M_{p}$ and $M_{s}$ are set on top of $M_{h}$, as illustrated in Figure 1. In AMTL, the error is reversely propagated from $M_{s}$ to $M_{h}$ such that the output layer of $M_{h}$ is forced to learn speaker-invariant features so as to confuse $M_{s}$, while $M_{s}$ tries to correctly classify outputs of $M_{h}$ into their corresponding speakers. At the same time, $M_{p}$ learns to predict the correct DPGMM labels of input features, and back-propagate errors to $M_{h}$ in a usual way.

Let $\theta_{p},\theta_{s}$ and $\theta_{h}$ denote the network parameters of $M_{p},M_{s}$ and $M_{h}$, respectively. With the stochastic gradient descent (SGD) algorithm, these parameters are updated as,

[TABLE]

where $\delta$ is the learning rate, $\lambda$ is the adversarial weight, $\mathcal{L}_{p}$ and $\mathcal{L}_{s}$ are the loss values of subword and speaker classification tasks respectively, both in terms of cross-entropy. To implement Eqt. (4), a gradient reversal layer (GRL) [20] was designed to connect $M_{h}$ and $M_{s}$. The GRL acts as identity transform during forward-propagation and changes the sign of loss during back-propagation. After training, the output of $M_{h}$ is speaker-invariant and subword discriminative bottleneck feature (BNF) representation of input speech. Besides, the softmax output representation of $M_{p}$ is believed to carry less speaker information than that without performing speaker adversarial training.

2.4 Subword unit inference and smoothing

Subword unit sequences for the concerned untranscribed speech utterances are inferred from softmax PG representation of $M_{p}$ in the speaker AMTL DNN. For each input frame to the DNN, the DPGMM label with the highest probability in PG representation is regarded as the subword unit assigned to this frame. These frame-level unit labels are further processed by collapsing consecutive repetitive labels to form pseudo transcriptions.

We observed non-smoothness in the inferred unit sequences by using the above methods, i.e., frame-level unit labels that are isolated without temporal repetition. Considering that ground-truth phonemes generally span at least several frames, these non-smooth labels are unwanted. This work proposes an empirical method to filter out part of the non-smooth unit labels, which is summarized in Algorithm 1.

3 ZeroSpeech 2017 experiments

3.1 Dataset and evaluation metric

ZeroSpeech 2017 development dataset consists of three languages, i.e. English, French and Mandarin. Speaker information for training sets are given while unknown for test sets. The durations of training sets are $45,24$ and $2.5$ hours respectively. Detailed information of the dataset can be found in [2].

The evaluation metric is ABX subword discriminability. Basically, it is to decide whether $X$ belongs to $x$ or $y$ if $A$ belongs to $x$ and $B$ belongs to $y$, where $A,B$ and $X$ are speech segments, $x$ and $y$ are two phonemes that differ in the central sound (e.g., “beg”-“bag”). Each pair of $A$ and $B$ is spoken by the same speaker. Depending on whether $X$ and $A(B)$ are spoken by the same speaker, ABX error rates for across-/within-speaker are evaluated separately. Dynamic time warping and cosine distance are used to measure segment- and frame-level dissimilarity.

3.2 System setup

The FHVAE model is trained with merged training sets of all three target languages. Input features are fixed-length speech segments of $10$ frames. Each frame is represented by a $13$-dimensional MFCC with cepstral mean normalization (CMN) at speaker level. During training, speech utterances spoken by the same speaker are concatenated to a single training sequence. During the inference of hidden variables $\bm{z_{1}}$ and $\bm{z_{2}}$, input segments are shifted by $1$ frame. To match the length of latent variables with original features, the first and last frame are padded. To generate speaker-invariant reconstructed MFCCs using the s-vector unification method, a representative speaker is selected from training sets. In this work the English speaker ‘s4018’ is chosen. The encoder and decoder networks of the FHVAE are both $2$-layer LSTM with $256$ neurons per layer. Latent variable dimensions for $\bm{z_{1}}$ and $\bm{z_{2}}$ are $32$. FHVAE training is implemented in Tensorflow [25] with the Adam [26] optimizer, using tools developed by [18].

The FHVAE based speaker-invariant MFCC features with $\Delta$ and $\Delta\Delta$ are fed as inputs to DPGMM clustering. Training data for the three languages are clustered separately. The numbers of clustering iterations for English, French and Mandarin are $80,80$ and $1400$. After clustering, the numbers of clusters are $591,526$ and $314$. Frame labels are used as pseudo phone alignments to support multilingual DNN training. Speaker labels obtained from training data support the adversarial speaker classification task. DNN input features are MFCC+CMVN. The layer-wise structure of $M_{h}$ is $\{1024\times 5,40\}$. Nonlinear function is sigmoid, except the linear BN layer. $M_{s}$ contains $3$ sub-networks, one for each language. The sub-network contains a GRL, a feed-forward layer (FFL) and a softmax layer. The GRL and FFL are $1024$-dimensional. $M_{p}$ also contains $3$ sub-networks, each having a $1024$-dimensional FFL and a softmax layer. During AMTL DNN training, the learning rate starts from $8\cdot 10^{-3}$ to $8\cdot 10^{-4}$ with exponential decay. The number of epochs is $5$. Speaker adversarial weight $\lambda$ ranges from [math] to $0.1$. After training, BNFs extracted from $M_{h}$ are evaluated by the ABX task. DNN is implemented using Kaldi [27] nnet3 recipe. DPGMM is implemented using tools developed by [19].

In addition to DPGMM clustering by FHVAE based speaker-invariant features, we also implement clustering by raw MFCC features and generate alternative DPGMM labels for comparison. In this case, the numbers of clustering iterations for the three languages are $120,200$ and $3000$. The numbers of clusters are $1118,1345$ and $596$. The DNN structure and training procedure are the same as mentioned above.

3.3 Experimental results

Average ABX error rates on BNFs over three target languages with different values of $\lambda$ are shown in Figure 2.

In this Figure, $\lambda=0$ denotes that speaker adversarial training is not applied. From the dashed (blue) lines, it can be observed that speaker adversarial training could reduce ABX error rates in both across- and within-speaker conditions, with absolute reductions of $0.28\%$ and $0.23\%$ respectively. The amount of improvement is in accordance with the findings reported in [17], despite that [17] exploited English transcriptions during training. The dash-dotted (red) lines show that when DPGMM labels generated by reconstructed MFCCs are employed in DNN training, the positive impact of speaker adversarial training in across-speaker condition is relatively limited. Besides, negative impact is observed in within-speaker condition. From Figure 2, it can be concluded that for the purpose of improving the robustness of subword modeling towards speaker variation, frame labeling based on disentangled speech representation learning is more prominent than speaker adversarial training.

4 ZeroSpeech 2019 experiments

4.1 Dataset and evaluation metrics

ZeroSpeech 2019 [3] provides untranscribed speech data for two languages. English is used for development while the surprise language (Indonesian) [28, 29] is used for test only. Each language pack consists of training and test sets. The training set consists of a unit discovery dataset for building unsupervised subword models, and a voice dataset for training the TTS system. The optional parallel dataset is not used in this work. Details of ZeroSpeech 2019 datasets are listed in Table 1.

There are two categories of evaluation metrics in ZeroSpeech 2019. The metrics for text embeddings, i.e. automatically discovered subword unit sequences as well as feature representations e.g. BNFs or PGs, are ABX discriminability and bitrate. ABX error rate is the same as in ZeroSpeech2017. Bitrate is defined as the amount of information provided in the inferred unit sequences. The metrics for synthesized speech waveforms are character error rate (CER), speaker similarity (SS, $1$ to $5$, larger is better) and mean opinion score (MOS, $1$ to $5$, larger is better), all evaluated by native speakers.

4.2 System setup

FHVAE model training and speaker-invariant MFCC reconstruction are performed following the configurations in ZeroSpeech 2017. The unit dataset is used for training. During MFCC reconstruction, a male speaker for each of the two languages is randomly selected as the representative speaker for s-vector unification. Our recent research findings [11] showed that male speakers are more suitable than females in generating speaker-invariant features. The IDs of the selected speakers are ‘S015’ and ‘S002’ in English and Surprise respectively. In DPGMM clustering, the numbers of clustering iterations are both $320$. Input features are reconstructed MFCCs+$\Delta$+$\Delta\Delta$. After clustering, the numbers of clusters are $518$ and $693$. The speaker AMTL DNN structure and training procedure follow configurations in ZeroSpeech 2017. One difference is the placement of adversarial sub-network $M_{s}$. Here $M_{s}$ is put on top of the FFL in $M_{p}$ instead of on top of $M_{h}$. Besides, the DNN is trained in a monolingual manner. After DNN training, PGs for voice and test sets are extracted. BNFs for test set are also extracted. Adversarial weights $\lambda$ ranging from [math] to $0.12$ with a step size of $0.02$ are evaluated on English test set.

The TTS model is trained with voice dataset and their subword unit sequences inferred from PGs. TTS training is implemented using tools [30] in the same way as in the baseline. The trained TTS synthesizes speech waveforms according to unit sequences inferred from test speech utterances. Algorithm 1 is applied to voice set and optionally applied to test set.

4.3 Experimental results

ABX error rates on subword unit sequences, PGs and BNFs with different values of $\lambda$ evaluated on English test set are shown in Figure 3. Algorithm 1 is not applied at this stage. It is observed that speaker adversarial training could achieve $0.49\%$ and $0.30\%$ absolute error rate reductions on PG and BNF representations. The unit sequence representation does not benefit from adversarial training. Therefore, the optimal $\lambda$ for unit sequences is [math]. The performance gap between frame-level PGs and unit sequences measures the phoneme discriminability distortion caused by the unit inference procedure in this work.

We fix $\lambda=0$ to train the TTS model, and synthesize test speech waveforms using the trained TTS. Experimental results of our submission systems are summarized in Table 2.

In this Table, ‘+SM’ denotes applying sequence smoothing towards test set unit labels. Compared with the official baseline, our proposed approaches could significantly improve unit quality in terms of ABX discriminability. Our system without applying SM achieves $14.6\%$ and $16.8\%$ absolute error rate reductions in English and Surprise sets. If SM is applied, while the ABX error rate increases, improvements in all the other evaluation metrics are observed. This implies that for the goal of speech synthesis, there is a trade off between quality and quantity of the learned subword units. Besides, our ABX performance is competitive to, or even better than the supervised topline.

Our systems do not outperform baseline in terms of synthesis quality. One possible explanation is that our learned subword units are much more fine-grained than those in the baseline AUD, making the baseline TTS less suitable for our AUD system. In the future, we plan to investigate on alternative TTS models to take full advantage of our learned subword units.

5 Conclusions

This study tackles robust unsupervised subword modeling in the zero-resource scenario. The robustness towards speaker variation is achieved by combining speaker adversarial training and FHVAE based disentangled speech representation learning. Our proposed approaches are evaluated on ZeroSpeech 2019 and ZeroSpeech 2017. Experimental results on ZeroSpeech 2017 show that both approaches are effective while the latter is more prominent, and that their combination brings further marginal improvement in across-speaker condition. Results on ZeroSpeech 2019 show that our approaches achieve significant ABX error rate reduction to the baseline system. The proposed unit sequence smoothing algorithm improves synthesis quality, at a cost of slight decrease in ABX discriminability.