Language Modeling with Deep Transformers

TL;DR

This paper demonstrates that deep autoregressive Transformer models, configured specifically for language modeling, outperform LSTM baselines in speech recognition tasks and can operate effectively without positional encoding due to inherent positional information in autoregressive setups.

Contribution

The study shows that well-configured deep Transformer language models outperform LSTM baselines and can function without positional encoding in autoregressive language modeling.

Findings

Deep Transformers outperform LSTM baselines in speech recognition.

Removing positional encoding can slightly improve deep autoregressive Transformer performance.

Deep autoregressive models utilize inherent positional information without explicit encoding.

Abstract

We explore deep autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.

Figures

Tables

Table 1. Table 1: Perplexities of the baseline models.

Model	Drop-	Bottle-	Num.	Num.	Params	Dev	Test
	out	beck	units	layers	in M
4-gram	-	-	-	-	230	146.2	151.8
LSTM	0.2	None	2048	2	487	71.3	74.8
	0.0					66.6	69.9
				3	520	64.0	67.2
				4	554	61.9	64.9
				5	587	62.7	65.9
				6	621	64.5	67.5
				8	688	67.2	70.3
			4096	2	1048	60.2	63.2
		512			334	63.1	66.3
			2048	4	248	64.5	67.7

Table 2. Table 2: Perplexity after 2.5 epoch (25 sub-epochs in our setup; 6.5M updates). The number of heads H 𝐻 H is 8 for all models below.

Input	$L$	$d_{ff}$	$d_{res}$	Params.	Perplexity
emb.				in M	Train	Dev
512	1	2048	512	208	108.3	104.9
	6			224	75.7	74.3
	12			243	67.6	67.1
	24			281	62.2	62.3
	32			306	60.1	60.6
	42			338	59.0	59.6
512	2	8192	2048	536	73.1	73.8
	6		512	262	66.7	66.7
	12	4096		268	63.5	63.8
	4	16384		277	67.6	67.4
		32768		344	65.4	68.4
128	64	2048	512	330	56.3	57.6
	80			380	53.1	55.5
	96			431	51.9	54.9
	112			481	51.5	54.5

Table 3. Table 3: Effect of number of heads. Perplexity after 2.5 epoch for (12, 2048, 512, H 𝐻 H ).

$H$	Params.	Perplexity
	in M	Train	Dev
1	243	71.9	70.8
4		69.1	68.6
8		67.6	67.1
16		66.9	66.6

Table 4. Table 5: Perplexities after longer training .

Max.	Conv-	$L$	$d_{ff}$	$d_{res}$	Params.	Perplexity
Epoch	erged				in M	Train	Dev	Test
5.5	Yes	12	4096	512	268	57.3	59.9	62.3
5		24	2048		281	55.6	58.0	60.7
		32			306	53.4	56.6	59.5
		42			338	51.2	55.0	57.7
3	No	80	2048	512	380	51.9	54.3	56.9
		96			431	50.9	53.7	56.3

Table 5. Table 6: Perplexity after 2.5 epoch for ( L 𝐿 L , 8192, 1024, 16) models with shared parameters across all layers.

$L$	Params.	Perplexity
	in M	Train	Dev
3	329	82.6	79.9
6		76.7	74.6
12		74.2	72.1

Table 6. Table 7: WERs (%) for hybrid systems on the LibriSpeech 960hr . 4-gram model is used in the first pass to generates lattices for rescoring. The row ”Lattice” shows oracle WERs of the lattices.

LM	$L$	Para.	dev				test
		in M	clean		other		clean		other
			PPL	WER	PPL	WER	PPL	WER	PPL	WER
4-gram	-	230	151.7	3.4	140.6	8.3	158.1	3.8	145.7	8.8
Lattice	-	-	-	1.0	-	2.3	-	1.3	-	2.6
LSTM	2	1048	60.2	2.3	60.2	5.4	64.8	2.6	61.7	5.9
Trans-	24	281	57.8	2.2	58.3	5.2	62.2	2.5	59.4	5.7
former	42	338	54.5	2.1	55.5	5.2	59.1	2.5	56.4	5.7
	96	431	53.2	2.1	54.2	5.2	57.6	2.5	55.0	5.6

Table 7. Table 8: WERs (%) for attention-based models on LibriSpeech 960hr dataset. Perplexities are on the 10K BPE level.

LM	Beam	dev				test
		clean		other		clean		other
		PPL	WER	PPL	WER	PPL	WER	PPL	WER
None	12	-	4.3	-	12.9	-	4.4	-	13.5
LSTM	64	43.7	2.9	46.4	8.9	47.1	3.2	47.2	9.9
Transfo.		35.9	2.6	38.9	8.4	38.8	2.8	39.0	9.3

Table 8. Table 9: Effect of sinusoidal positional encoding. Perplexity after 5 epochs (13M updates) for ( L 𝐿 L , 2048, 512, 8) models.

$L$	Position.	Params.	Perplexity
	encoding	in M.	Train	Dev	Test
12	Sinusoidal	243	61.8	63.1	66.1
	None		58.0	60.5	63.4
24	Sinusoidal	281	55.6	58.0	60.8
	None		52.7	56.6	59.2
42	Sinusoidal	338	51.2	55.0	57.7
	None		50.5	54.2	56.8

Equations

$$x_t^{(l)}$$

$$q_t^{(l)},k_t^{(l)},v_t^{(l)}$$

$$h_t^{(l)}$$

$$y_t^{(l)}$$

$$m_t^{(l)}$$

$$z_t^{(l)}$$

Full text

Index Terms: language modeling, self-attention, Transformer, speech recognition

1 Introduction

Transformer encoder-decoder models [1] have become popular in natural language processing. The Transformer architecture allows to successfully train a deep stack of self-attention layers [2, 3, 4] via residual connections [5] and layer normalization [6]. The positional encodings [1, 7], typically based on sinusoidal functions, are used to provide the self-attention with the sequence order information. Across various applications, systematic improvements have been reported over the standard, multi-layer long short-term memory (LSTM) [8] recurrent neural network based models. While originally designed as an encoder-decoder architecture in machine translation, the encoder (e.g., [9]) and the decoder (e.g., [10]) components are also separately used in corresponding problems depending on whether the problem disposes the whole sequence for prediction or not.

A number of recent works have also shown impressive performance in language modeling using the Transformer decoder component [10, 11, 12, 13, 14, 15]. The earliest example can be found in [10] where such models are investigated for text generation. Recent works on training larger and deeper models [12, 14, 15] have shown further potential of the Transformer in language modeling. On the other hand, an obvious limitation of the Transformers is that their memory requirement linearly increases in terms of number of tokens in the sequence, which requires to work with a limited context window (basically a $n$-gram model where the typical number for $n$ is 512) for tasks dealing with long sequences such as character-level language modeling [12]. Dai et al. [11] has introduced a segment-level recurrence and relative positional encoding in the Transformer language model to be able to potentially handle unlimited context.

In this work, we investigate deep autoregressive Transformers for language modeling in speech recognition. To be specific, we focus on two aspects. First, we revisit the parameter configurations of Transformers, originally engineered for the sequence-to-sequence problem [1], specifically for language modeling. We conduct experiments on the LibriSpeech automatic speech recognition (ASR) task [16] for both word-level conventional speech recognition and byte-pair encoding (BPE) [17] level end-to-end speech recognition [18, 19]. We apply our word-level models to hybrid speech recognition by lattice rescoring [20], and the BPE-level models to end-to-end models by shallow fusion [21, 22]. We show that well configured Transformer language models outperform models based on the simple stack of LSTM RNN layers in terms of both perplexity and word error rate (WER).

Second, we experimentally show that the positional encoding is not needed for multi-layer autoregressive self-attention models. The visualization of the attention weights shows that when the sinusoidal positional encoding is provided with the input, the first layer of the Transformers learns to extract $n$-gram features (therefore making use of positional information). However, in the autoregressive problem where a new token is provided to the model at each time step, the amount of information the model has access to strictly increases from left to right at the lowest level of the network, which should provide some positional information by its own. We observe that deep Transformer language models without positional encoding automatically make use of such information, and even give slight improvements over models with positional encodings.

2 Related Work

The first part of our work follows the spirits of Al-Rfou et al.’s work [12] and Radford et al.’s work [14, 15] in investigating larger and deeper Transformers for language modeling. We show that deep Transformer language models can be successfully applied to speech recognition and give good performance. The second part of this work concerns the positional encoding, which is a crucial component in the original Transformer. A number of previous work investigated positional encoding variants to improve self-attention (e.g., [23, 24, 25, 11]). Previous works in Transformer language models systematically use positional encoding, either jointly learned one or the sinusoidal one (both cases are reported to give similar performance in [12]). We show that the deep autoregressive self-attention models do not require any explicit model for encoding positions to give the best performance.

3 Autoregressive Self-Attention

The language model we consider is based on the decoder component of the Transformer architecture [1]. Similar to previous work [10, 14, 13, 12, 11, 15], we define layer as a stack of two components: self-attention and feed-forward111Typically called position-wise feed-forward module [1]. Here we omit position-wise as it is obvious for autoregressive models. modules.

The autoregressive self-attention module in the $l$-th layer transforms the input $z_{t}^{(l-1)}$ at position $t$ as follows:

[TABLE]

where $Q$, $K$, $V$, respectively denote query, key, value projection matrices, $\operatorname{LayerNorm}$ denotes layer normalization [6], $\operatorname{SelfAttention}$ denotes the scaled multi-head dot product self-attention [1], and $W_{0}$ denotes the projection matrix for the residual connection [5].

The output $y_{t}^{(l)}$ is then fed to the feed-forward module:

[TABLE]

where $\operatorname{Activation}$ is rectifier [26], Gaussian error linear unit (GELU) [27, 15], or gated linear unit (GLU) [28] in this work. The final model is build by stacking these layers multiple times.

The input of the network consists of the sum of the token embedding (word or BPE in this work) and the sinusoidal positional encoding as specified in [1]. The output softmax layer gives the probability distribution for the next token. As shown in the equations above, $h_{t}^{(l)}$ can be seen as states of the Transformer model222In principle, we could also consider an autoregressive self-attention model which updates states at all predecessor positions for each new input, which would be then much more computationally inefficient. (whose size, as opposed to the RNN states, linearly grows along the position dimension). During inference, these states are stored to avoid redundant computation. During training, the computation along the position dimension is parallelized for speed-up.

4 LibriSpeech Dataset

4.1 Language Modeling Data Descriptions

The LibriSpeech datasets [16] for language modeling consists of 800M-word text only data and 960hr of audio transcriptions which corresponds to 10M-word text data. Based on analysis of count model perplexities, we observe that the audio transcription part does not contain special domain signal which matches the development set. Therefore, we simply merge the two datasets to form a single dataset for language model training. The average sentence length in the resulting training data is 21 words with the maximum length of 600 words. The development and test sets respectively have two parts[16]: dev-clean, dev-other, test-clean, and test-other. This separation is based on the audio-level characteristics, therefore it has no special meaning for language modeling. In the experimental section, we denote by ”Dev” and ”Test” the concatenation of clean and other parts of the respective data. Both datasets consist of about 110K running words with average of 20 words per sentence. The word-level vocabulary contains 200K words. We report all perplexities without making use of contexts beyond the sentence boundary.

4.2 4-gram count and LSTM-RNN Baselines

We use the official 4-gram count language model provided with the LibriSpeech dataset [16]. No improvement in perplexity is observed when going up to 5-grams. For LSTM-RNN language models [29], we first train our base configuration; the model has 2 LSTM-RNN layers with 2048 nodes and the input projection layer of 128, where the dropout with a rate of 0.2 is applied between each layer. Since we observe that this model underfits the LibriSpeech training set, we remove the dropout and further increase the model size, which effectively give better perplexities as shown in Table 1. We find that improvements from simply stacking layers saturate at 4 layers even without overfitting. Introducing a small linear bottleneck layer (size 512 here) before the output layer can make the models compact but with a loss in performance. The best model we obtain has 2 layers with 4096 nodes. Relative improvements greater than 58% are obtained by the LSTM over the 4-gram language model.

5 Text based Experiments

We carry out experiments for both word-level and BPE-level language modeling. We first focus on the word-level one.

5.1 Hyper-parameters in Transformers

The Transformer architecture is a new search space Odyssey [30]. The exhaustive model hyper-parameters for Transformer language models specified by the equations in Sec. 3 are the input token embedding size, the number of layers, the dimension of the residual connection, and for each layer the number of attention heads, the dimension of the key and query, the dimension of the value, and the dimension of the feed-forward layer.

In our experiments, we use the same dimension for key, query and value, as well as the residual connection. We use the same dimensionality across all layers. Therefore, our models can be fully specified by the tuple (number of layers $L$, feed-forward dimension $d_{ff}$, residual dimension $d_{res}$, number of heads $H$). We do not apply any regularization method including dropout. We train all models using the plain stochastic gradient descent and new-bob learning rate tuning on a single GPU. We define our training sub-epoch (for new-bob) as the 10th of the full training data. All our implementations are based on the Tensorflow [31] based open-source toolkit RETURNN [32]333Training configuration files and trained models are available at https://github.com/rwth-i6/returnn-experiments/tree/master/2019-lm-transformers..

5.2 Hyper-parameter Tuning

Given the amount of LibriSpeech training data (810M words), it is unreasonable to train all model variants until full convergence. The earlier stage of the training already consistently indicates the potential performance of the models. Therefore, we first carry out comparisons between models with different configuration at the equal, large enough, but reasonable number of updates.

The first set of comparison investigates the effect of depth and width. The perplexity results can be found in Table 2. All models in the table use 8 attention heads. Other parameters are specified in the table. The table is organized in three parts. The upper part of Table 2 shows the effect of number of layers; we observe that increasing number of layers (therefore the number of parameters) from 1 to 42 gradually improves the perplexity. In the middle part of Table 2 , we vary both the number of layers, feed-forward dimension, and the residual dimension. First of all, the 12-layer $(12,4096,512,8)$ model outperforms the 6-layer $(6,8192,512,8)$ model, while having similar number of parameters, which seems to indicate that the depth effectively benefits Transformer language models. We also train an extreme model which has only 2 layers with wide dimensions $(2,8192,2048,8)$. The number of parameters in fact blows up because of the large value of $d_{res}$ which results in a large matrix in the output softmax layer with 200K vocabulary444 We note that this is also the reason why the number of parameters of our baseline LSTM language models in Table 1 is relatively high.. We observe that such wide but shallow models do not perform well555Since the softmax bottleneck dimension typically needs to be large for the best performance ([33]; Table 1), we also train a (12, 2048, 512, 8) model where we insert an additional projection layer with a large dimension (2048) before the output layer; no improvement was obtained. . Finally, the lower part of Table 2 shows deeper models with a smaller input dimension.

Table 4 shows the effect of number of attention heads. 16 heads which is the largest number we try in this setup give the best performance. In addition, we examine the type of activation function (Table 4). As opposed to previous work on feed-forward language models using GLUs [28, 34], we do not observe faster convergence. As we observe that the impact of choice of activation functions on the perplexity is overall limited, all our other models use the standard ReLU. As reported in the original Transformer, we confirm that both layer normalization and residual connections are needed for these models for stable training666We tried to train multiple models without either residual connections or layer normalization. Also, following [15], we tried reorganizing the feed-forward module to insert one additional pre-activation layer normalization [35] and one more activation function. However, we did not observe any improvement. The original Transformers anyway do not have any activation on the residual path throughout the whole network..

Finally, we train models with the best configurations for longer. Table 5 shows the perplexities which are better than those obtained by our LSTM based models (Table 1).

5.3 Parameter Tying

Dehghani et al. [36] reports Universal Transformers to perform particularly well for language modeling. This motivates us to experiment with parameter sharing across layers. For such models to have comparable number of parameters with the standard deep Transformers, the dimensions in each layer must be increased, which results in slower training; here we simply investigate the effect of number of recurrence. Table 6 shows the perplexity results. First of all, we observe that the model performance is behind that of the standard Transformer777We note that here the direct comparison is not as straightforward as between the standard Transformers. In fact, we observe that the training hyperparameters tuned for the standard Transformers can not be directly applied to Universal Transformers; specifically, we find it crucial to reduce the gradient norm clipping threshold from 1 to 0.1, which is potentially slowing down the convergence. (Table 2). However, we clearly observe that increasing the number of layers from 3 to 12 consistently improves the perplexity. This improvement without additional parameters motivates future work to investigate further parameter sharing strategies for Transformers.

6 ASR Experiments

6.1 Lattice Rescoring Results

We apply our word-level Transformer language models to conventional hybrid speech recognition by lattice rescoring. The standard push-forward lattice rescoring algorithm [20] for long-span language models can be directly applied to self-attention based models. The only modifications from the RNN version is to define the ”state” as all hidden states ($h_{t}^{(l)}$ in Sec.3) in all layers from all predecessor positions and the current position ($t$; for position encoding). Table 7 shows the WERs and perplexities (PPL). Our baseline acoustic model is based on multi-layer bi-directional LSTM [37]. Further descriptions of our baseline acoustic model can be found in [38]. We obtain consistent improvements in terms of WER over the LSTM baselines.

6.2 End-to-End ASR Shallow Fusion Results

We train 10K BPE-level Transformer language models to be combined with an attention-based encoder-decoder speech model by shallow fusion [21, 22]. The 10K BPE level training data has a longer average length of 24 tokens per sentence with the longest sentence length of 1343, which is still manageable without any truncation for self-attention. We use the Transformer architecture of (24, 4096, 1024, 8). The LSTM model has 4 layers with 2048 nodes. We refer to our previous work [19] for the description of the baseline attention model; the baseline WERs better than our previous work [19] are obtained by improved curriculum learning and longer training. Table 8 shows both perplexities and WERs. Following [39], we introduce an end-of-sentence penalty in shallow fusion to benefit from a large beam size of 64. Again, we obtain consistent improvements over the LSTM baseline. These results are better than previously reported WERs [39, 40, 41] for end-to-end models without data augmentation [42].

7 Analysis

Compared with hidden states in RNNs, attention weights are easier to be visualized, which gives opportunity for analysis. In particular, we focus on the comparison of the Transformer language models with and without positional encoding.

7.1 Transformer LM without positional encoding

In the autoregressive problem where a new token is provided to the model at each time step, the amount of information the model has access to strictly increases from left to right at the lowest level of the network; the deeper layers should be able to recognize this structure which should provide the model with some positional information by its own. To check this hypothesis, we train models without any positional encoding. First, we observe that they give better perplexities than the models with sinusoidal positional encoding (Table 9).

7.2 First layer

The attention in the first layer is the most straightforward for interpretation because the feature at each position exactly corresponds to the word at the position (while deeper layers can potentially shuffle the feature content). The attention weights in the first layer of 24-layer Transformer language models with and without positional encodings are visualized in Figure 1. We observe that the first layer of the model with positional encoding (Figure 1(a)) learns to create n-gram features (roughly 2 or 3-gram), which indicates that the positional information is directly used. In contrast, the first layer of the model without positional encoding learns to focus on the new input token as can be seen as the diagonal in Figure 1(b) (interestingly, we also see that it ignores some functional words such as ”the”, ”and”, ”to” which might be modeled by some off-set values, therefore attending to the beginning of sentence token instead), which demonstrates that the model is aware of the position of the new input.

7.3 Other layers

We observe that the behavior of other layers are rather similar for both Transformer models with and without positional encoding. We find 3 categories of layers in the other 23 layers; the second and third layers are ”blur” layers as shown in Figure 1(c), which seems to roughly average over all positions (while we can also see that some heads focus on difficult words, here ”verandah”). Layer 4 to 9 are window layers which focus on the local n-gram. A representative example is show in Figure 1(d). Finally, we find the top layers 10 to 24 to be more structured, attending to some specific patterns; an example is shown in Figure 1(e).

8 Conclusion

We apply deep Transformer language models for speech recognition. We show that such models outperform the shallow stack of LSTM-RNNs on both word-level and BPE-level modeling. Future work investigates application of crucial components of deep Transformers (such as layer normalization) to deeper LSTM models; e.g., the RNMT+ decoder architecture [43] for language modeling. Furthermore, we do not apply any regularization on models for the LibriSpeech task, as no overfitting is observed in the range of model sizes we experimented with (for the word-level models). We can possibly still improve our models simply by scaling up their size and using regularization.

Acknowledgements

This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 694537, project ”SEQCLAS”) and from a Google Focused Award. The work reflects only the authors’ views and none of the funding parties is responsible for any use that may be made of the information it contains. We thanks Liuhui Deng for contributing to our lattice rescoring code, Arne Nix and Julian Schamper for sharing their base Transformer configs, as well as Eugen Beck, Christoph Lüscher and Wei Zhou for help with generating lattices. Experiments were partially performed with computing resources granted by RWTH Aachen University under project nova0003.