Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

TL;DR

The paper introduces Federation of Experts (FoE), a novel architecture that reduces communication bottlenecks in distributed large language models, significantly improving inference speed while maintaining quality.

Contribution

FoE restructures MoE layers to minimize communication overhead, enabling efficient inference in distributed LLMs with up to 5.2x latency reduction.

Findings

FoE reduces end-to-end latency by up to 5.2x.

FoE maintains comparable generation quality to traditional MoE models.

FoE significantly improves throughput and latency in both single-node and multi-node settings.

Abstract

Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TBT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.