Compact Recurrent Transformer with Persistent Memory

TL;DR

This paper introduces a Compact Recurrent Transformer (CRT) that efficiently processes long sequences by combining shallow local transformers with a persistent memory, reducing computational costs while maintaining high performance.

Contribution

The novel CRT model integrates recurrent neural networks with shallow transformers to effectively manage global information with lower computational overhead.

Findings

CRT achieves comparable or better results than full Transformers on language tasks.

CRT uses significantly fewer FLOPs and shorter segments.

State-of-the-art performance on Toyota Smarthome dataset.

Abstract

The Transformer architecture has shown significant success in many language processing and visual tasks. However, the method faces challenges in efficiently scaling to long sequences because the self-attention computation is quadratic with respect to the input length. To overcome this limitation, several approaches scale to longer sequences by breaking long sequences into a series of segments, restricting self-attention to local dependencies between tokens within each segment and using a memory mechanism to manage information flow between segments. However, these approached generally introduce additional compute overhead that restricts them from being used for applications where limited compute memory and power are of great concern (such as edge computing). We propose a novel and efficient Compact Recurrent Transformer (CRT), which combines shallow Transformer models that process short local segments with recurrent neural networks to compress and manage a single persistent memory vector that summarizes long-range global information between segments. We evaluate CRT on WordPTB and WikiText-103 for next-token-prediction tasks, as well as on the Toyota Smarthome video dataset for classification. CRT achieves comparable or superior prediction results to full-length Transformers in the language datasets while using significantly shorter segments (half or quarter size) and substantially reduced FLOPs. Our approach also demonstrates state-of-the-art performance on the Toyota Smarthome video dataset.