Bird-Eye Transformers for Text Generation Models

TL;DR

This paper introduces the bird-eye transformer (BET), a novel architecture that enhances text generation by reweighting self-attention to better incorporate historical information, outperforming baseline transformers across multiple datasets.

Contribution

The paper proposes BET, a new transformer variant that improves historical information utilization in text generation tasks, addressing limitations of standard self-attention.

Findings

BET outperforms baseline transformers on all tested datasets.

Reweighting self-attention improves historical context integration.

Experimental results demonstrate superior performance in machine translation and language modeling.

Abstract

Transformers have become an indispensable module for text generation models since their great success in machine translation. Previous works attribute the~success of transformers to the query-key-value dot-product attention, which provides a robust inductive bias by the fully connected token graphs. However, we found that self-attention has a severe limitation. When predicting the (i+1)-th token, self-attention only takes the i-th token as an information collector, and it tends to give a high attention weight to those tokens similar to itself. Therefore, most of the historical information that occurred before the i-th token is not taken into consideration. Based on this observation, in this paper, we propose a new architecture, called bird-eye transformer(BET), which goes one step further to improve the performance of transformers by reweighting self-attention to encourage it to focus more on important historical information. We have conducted experiments on multiple text generation tasks, including machine translation (2 datasets) and language models (3 datasets). These experimental~results show that our proposed model achieves a better performance than the baseline transformer architectures on~all~datasets. The code is released at: \url{https://sites.google.com/view/bet-transformer/home}.