Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networks

TL;DR

This paper introduces a novel deep Transformer architecture that re-examines input features at multiple depths using intermediate losses, leading to significant performance improvements in speech and video recognition tasks.

Contribution

It proposes a double feature presentation with iterated loss in deep Transformers, enabling input feature re-use and improved accuracy over standard models.

Findings

10-20% relative improvement on Librispeech

3.2-13% relative improvement on video dataset

Iterated loss enhances model performance and robustness

Abstract

Deep acoustic models typically receive features in the first layer of the network, and process increasingly abstract representations in the subsequent layers. Here, we propose to feed the input features at multiple depths in the acoustic model. As our motivation is to allow acoustic models to re-examine their input features in light of partial hypotheses we introduce intermediate model heads and loss function. We study this architecture in the context of deep Transformer networks, and we use an attention mechanism over both the previous layer activations and the input features. To train this model's intermediate output hypothesis, we apply the objective function at each layer right before feature re-use. We find that the use of such iterated loss significantly improves performance by itself, as well as enabling input feature re-use. We present results on both Librispeech, and a large scale video dataset, with relative improvements of 10 - 20% for Librispeech and 3.2 - 13% for videos.