Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems

TL;DR

This paper systematically analyzes Attention-FFN Disaggregation (AFD) in large-scale MoE models, revealing its performance limits on standard clusters and identifying hardware-model conditions where AFD is advantageous.

Contribution

It extends the roofline model to communication levels, providing insights into AFD's performance boundaries and conditions for effective deployment.

Findings

AFD faces a dead zone on standard clusters due to bandwidth limitations.

Higher imbalance penalties occur with discrete node-level scaling in AFD.

A specific hardware-model combination can mitigate AFD's limitations and enhance performance.

Abstract

Deploying large-scale MoE models presents challenges in memory capacity and bandwidth for expert activation. While Attention-FFN Disaggregation (AFD) has emerged as a potential architecture to decouple compute and memory resources, its performance boundaries compared to standard large-scale Expert Parallelism (EP) remain underexplored. In this paper, we conduct a systematic analysis of AFD by extending the roofline model to the communication level, correlating interconnect bandwidth, arithmetic intensity, and Hardware FLOPS Utilization (HFU). Our analysis reveals a dead zone on standard clusters: increasing FFN instance count fails to improve HFU as computational workload is capped by scale-out bandwidth, causing operator active time to shrink relative to the fixed latency budget. We further show that AFD's discrete node-level scaling incurs higher imbalance penalties than EP's continuous batch adjustment. Nevertheless, these limitations diminish under specific conditions: Superpod-class hardware with abundant interconnect bandwidth and models with coarse-grained experts and lower sparsity are more likely to benefit from AFD. These findings position AFD as a promising approach for specific hardware-model combinations rather than a universal solution.