FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models

TL;DR

FarSkip-Collective is a novel architectural modification that enables overlapping communication with computation in large Mixture of Experts models, significantly improving training and inference efficiency without sacrificing accuracy.

Contribution

The paper introduces FarSkip-Collective, a method to modify model architecture for overlapping communication and computation, maintaining accuracy in large models from 16B to 109B parameters.

Findings

Achieves near-original accuracy in large models after modification.

Enables overlapping communication with computation, accelerating training and inference.

Successfully applied to models like Llama 4 Scout (109B).

Abstract

Blocking communication presents a major hurdle in running MoEs efficiently in distributed settings. To address this, we present FarSkip-Collective which modifies the architecture of modern models to enable overlapping of their computation with communication. Our approach modifies the architecture to skip connections in the model and it is unclear a priori whether the modified model architecture can remain as capable, especially for large state-of-the-art models and while modifying all of the model layers. We answer this question in the affirmative and fully convert a series of state-of-the-art models varying from 16B to 109B parameters to enable overlapping of their communication while achieving accuracy on par with their original open-source releases. For example, we convert Llama 4 Scout (109B) via self-distillation and achieve average accuracy within 1% of its instruction tuned release averaged across a wide range of downstream evaluations. In addition to demonstrating retained accuracy of the large modified models, we realize the benefits of FarSkip-Collective through optimized implementations that explicitly overlap communication with computation, accelerating both training and inference in existing frameworks.