TreeCoders: Trees of Transformers

TL;DR

TreeCoders introduces a novel tree-structured transformer architecture that improves efficiency and flexibility, enabling sparse activation and distributed implementation, with competitive performance across language datasets.

Contribution

It presents a new tree-based transformer design with external selectors and sparse activation, differing from traditional linear transformers.

Findings

Outperforms size-matched linear transformers 76% of the time.

Supports sparse node activation with logarithmic complexity.

Demonstrates competitive results on diverse language datasets.

Abstract

In this paper, we introduce TreeCoders, a novel family of transformer trees. We moved away from traditional linear transformers to complete k-ary trees. Transformer blocks serve as nodes, and generic classifiers learn to select the best child and route the sequence of tokens to a specific leaf. The selectors, moved outside the transformer blocks, allow for the use of a variety of architecture without further modifications. Furthermore, our proposed architecture supports sparse node activation due to the logarithmic complexity of a tree search. We validate our idea by testing a series of decoder-only tree transformers, achieving competitive results across a diverse range of language datasets. Our study demonstrates that the proposed tree transformer model outperforms a size-equivalent linear transformer model 76\% of the time over a wide range of tree architectures. Furthermore, our proposed model naturally lends itself to distributed implementation.