ReviveMoE: Fast Recovery for Hardware Failures in Large-Scale MoE LLM Inference Deployments

TL;DR

ReviveMoE introduces a rapid failure recovery method for large-scale MoE LLM inference that avoids costly restarts, supporting both traditional and disaggregated architectures, and is integrated into Huawei Cloud's MaaS platform.

Contribution

It presents ReviveMoE, a novel recovery technique enabling fast failure handling in large-scale MoE LLM deployments without system restarts.

Findings

Supports both traditional and disaggregated architectures

Integrated into Huawei Cloud's MaaS platform

Enables rapid recovery without significant delays

Abstract

As LLM deployments scale over more hardware, the probability of a single failure in a system increases significantly, and cloud operators must consider robust countermeasures to handle these inevitable failures. A common recovery approach is to simply restart the LLM serving instance; however, this is costly in model-as-a-service (MaaS) inference settings, where reloading model weights and recompiling computation graphs can introduce significant delays to incoming requests. We propose ReviveMoE, a method for rapid failure recovery in large-scale LLM deployments without restarting the serving instance. ReviveMoE is designed to support both the traditional LLM architecture, which collocates MoE and attention on the same hardware, and the disaggregated architectures, which separate MoE from attention. Integrated into Huawei Cloud's MaaS, ReviveMoE is built on top of Huawei's xDeepServe serving platform and the XCCL communications library.