DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence

TL;DR

The paper introduces DCT, a transformer framework that efficiently models unbounded sequences by selectively retaining compressed sentence representations, outperforming previous models on the Enwik8 benchmark.

Contribution

It presents a novel memory management policy for transformers that improves handling of unlimited long sequences by selective compression and retention.

Findings

DCT outperforms previous SOTA on Enwik8.

Selective memory retention improves sequence modeling.

Compressed memory maintains semantic information effectively.

Abstract

In this paper, we propose Dynamic Compressive Transformer (DCT), a transformer-based framework for modeling the unbounded sequence. In contrast to the previous baselines which append every sentence representation to memory, conditionally selecting and appending them is a more reasonable solution to deal with unlimited long sequences. Our model uses a policy that determines whether the sequence should be kept in memory with a compressed state or discarded during the training process. With the benefits of retaining semantically meaningful sentence information in the memory system, our experiment results on Enwik8 benchmark show that DCT outperforms the previous state-of-the-art (SOTA) model.