Bidirectional Awareness Induction in Autoregressive Seq2Seq Models

TL;DR

This paper introduces Bidirectional Awareness Induction (BAI), a training method that enhances autoregressive seq2seq models by enabling bidirectional learning through Pivots, improving performance across multiple architectures and tasks.

Contribution

The paper presents BAI, a novel training approach that allows bidirectional learning in autoregressive models without architectural changes, applicable to various architectures and pre-trained models.

Findings

Up to 2.4 CIDEr improvement in Image-Captioning

Up to 4.96 BLEU increase in Neural Machine Translation

Up to 1.16 ROUGE boost in Text Summarization

Abstract

Autoregressive Sequence-To-Sequence models are the foundation of many Deep Learning achievements in major research fields such as Vision and Natural Language Processing. Despite that, they still present significant limitations. For instance, when errors occur in the early steps of the prediction, the whole output is severely affected. Such reliance on previously predicted tokens and the inherent computational unfriendliness of sequential algorithms, motivated researchers to explore different architectures and methods in the search for bidirectional approaches. In this work, we introduce the Bidirectional Awareness Induction (BAI), a training method that leverages a subset of elements in the network, the Pivots, to perform bidirectional learning without breaking the autoregressive constraints. To showcase its flexibility, we apply the method to three architectures, the Transformer, ExpansionNet v2 and GPT, then perform experiments over three tasks. Experimental results showcase BAI's effectiveness on all selected tasks and architectures. In particular, we observed an increase of up to 2.4 CIDEr in Image-Captioning, 4.96 BLEU in Neural Machine Translation, and 1.16 ROUGE in Text Summarization compared to the respective baselines. Notably, BAI not only has a positive impact on models trained from scratch but on pre-trained models as well. Such an aspect, combined with the absence of architectural requirements synergizes well with the current trend of LLMs.