ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive

TL;DR

ClusterFusion introduces cluster-level communication primitives and a scheduling framework to expand operator fusion in LLM inference, significantly reducing latency by enabling on-chip intermediate data processing.

Contribution

It proposes ClusterReduce and ClusterGather primitives for structured on-chip communication, enabling broader operator fusion in LLM inference on modern GPU architectures.

Findings

Achieves 1.61x average latency reduction over state-of-the-art frameworks.

Enables on-chip intermediate data exchange, reducing off-chip memory traffic.

Successfully fuses multiple decoding stages into single kernels.

Abstract

Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators and heavy reliance on off-chip memory for data exchange and reduction. This execution model limits opportunities for fusion and incurs significant memory traffic and kernel launch overhead. While modern architectures such as NVIDIA Hopper provide distributed shared memory and low-latency intra-cluster interconnects, they expose only low-level data movement instructions, lacking structured abstractions for collective on-chip communication. To bridge this software-hardware gap, we introduce two cluster-level communication primitives, ClusterReduce and ClusterGather, which abstract common communication patterns and enable structured, high-speed data exchange and reduction between thread blocks within a cluster, allowing intermediate results to be on-chip without involving off-chip memory. Building on these abstractions, we design ClusterFusion, an execution framework that schedules communication and computation jointly to expand operator fusion scope by composing decoding stages such as QKV Projection, Attention, and Output Projection into a single fused kernels. Evaluations on H100 GPUs show that ClusterFusion outperforms state-of-the-art inference frameworks by 1.61x on average in end-to-end latency across different models and configurations. The source code is available at https://github.com/xinhao-luo/ClusterFusion.