Two Heads Are Better than One: Simulating Large Transformers with Small Ones

TL;DR

This paper demonstrates that large transformers processing long sequences can be efficiently simulated by multiple small transformers, leveraging hardware optimization for short inputs, with theoretical bounds and practical scenarios analyzed.

Contribution

It introduces a method to simulate large transformers with small ones, providing theoretical bounds and practical scenarios where this approach is effective.

Findings

Large transformers can be simulated by O((N/M)^2) small transformers.

In natural scenarios, only O(N/M) small transformers are needed.

Theoretical bounds are proven to be optimal in the worst case.

Abstract

The quadratic complexity of self-attention prevents transformers from scaling effectively to long input sequences. On the other hand, modern GPUs and other specialized hardware accelerators are well-optimized for processing small input sequences in transformers during both training and inference. A natural question arises: can we take advantage of the efficiency of small transformers to deal with long input sequences? In this paper, we show that transformers with long input sequences (large transformers) can be efficiently simulated by transformers that can only take short input sequences (small transformers). Specifically, we prove that any transformer with input length $N$ can be efficiently simulated by only $O((N/M)^2)$ transformers with input length $M \ll N$, and that this cannot be improved in the worst case. However, we then prove that in various natural scenarios including average-case inputs, sliding window masking and attention sinks, the optimal number $O(N/M)$ of small transformers suffice.