Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

TL;DR

The paper introduces Mixture of Tokens (MoT), a continuous MoE architecture that scales parameters efficiently, is compatible with autoregressive tasks, and matches state-of-the-art performance while being faster to train.

Contribution

MoT is a novel continuous MoE design that assigns token mixtures across examples, enabling scalable, autoregressive-compatible models with competitive performance.

Findings

MoT achieves 3x faster training than dense Transformers.

MoT matches state-of-the-art MoE performance.

A new technique called transition tuning links MoT and traditional MoE.

Abstract

Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to substantially increase the parameter count without a corresponding increase in FLOPs. Most widely adopted MoE models are discontinuous with respect to their parameters - often referred to as sparse. At the same time, existing continuous MoE designs either lag behind their sparse counterparts or are incompatible with autoregressive decoding. Motivated by the observation that the adaptation of fully continuous methods has been an overarching trend in deep learning, we develop Mixture of Tokens (MoT), a simple, continuous architecture that is capable of scaling the number of parameters similarly to sparse MoE models. Unlike conventional methods, MoT assigns mixtures of tokens from different examples to each expert. This architecture is fully compatible with autoregressive training and generation. Our best models not only achieve a 3x increase in training speed over dense Transformer models in language pretraining but also match the performance of state-of-the-art MoE architectures. Additionally, a close connection between MoT and MoE is demonstrated through a novel technique we call transition tuning.