Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers

TL;DR

This paper introduces a simple, efficient framework for processing long sequences with transformers, reducing computational costs from quadratic to linear, and improving performance on long-text tasks.

Contribution

The authors propose a novel method that divides long sequences into chunks, aligns inter-chunk information, and selects key hidden states using a reinforcement learning-inspired policy.

Findings

Improved long-text summarization performance.

Reduced computational complexity from quadratic to linear.

Effective inter-chunk semantic alignment.

Abstract

Although dominant in natural language processing, transformer-based models remain challenged by the task of long-sequence processing, because the computational cost of self-attention operations in transformers swells quadratically with the input sequence length. To alleviate the complexity of long-sequence processing, we propose a simple framework to enable the offthe-shelf pre-trained transformers to process much longer sequences, while the computation and memory costs remain growing linearly with the input sequence lengths. More specifically, our method divides each long-sequence input into a batch of chunks, then aligns the interchunk information during the encoding steps, and finally selects the most representative hidden states from the encoder for the decoding process. To extract inter-chunk semantic information, we align the start and end token embeddings among chunks in each encoding transformer block. To learn an effective hidden selection policy, we design a dual updating scheme inspired by reinforcement learning, which regards the decoders of transformers as environments, and the downstream performance metrics as the rewards to evaluate the hidden selection actions. Our empirical results on real-world long-text summarization and reading comprehension tasks demonstrate effective improvements compared to prior longsequence processing baselines.