Fleet: Hierarchical Task-based Abstraction for Megakernels on Multi-Die GPUs

TL;DR

Fleet introduces a hierarchical task model for multi-die GPUs that improves cache utilization and reduces memory traffic, leading to significant performance gains in memory-bound workloads.

Contribution

The paper proposes Chiplet-tasks and a persistent kernel runtime to better map computation to GPU chiplet hierarchies, addressing limitations of current programming models.

Findings

Fleet achieves 1.3-1.5x lower decode latency than vLLM.

L2 hit rate increases from 12% to 54% at batch size 32.

Reduces HBM traffic by up to 37% and delivers 1.27-1.30x speedup.

Abstract

Modern GPUs adopt chiplet-based designs with multiple private cache hierarchies, but current programming models (CUDA/HIP) expose a flat execution hierarchy that cannot express chiplet-level locality or synchronization. This mismatch leads to redundant memory traffic and poor cache utilization in memory-bound workloads such as LLM inference. We present Fleet, a multi-level task model that maps computation to memory scopes. Fleet introduces Chiplet-tasks, a new abstraction that binds work and data to a chiplet and enables coordination through its shared L2 cache. Wavefront-level, CU-level, and device-level tasks align with existing abstractions, while Chiplet-tasks expose a previously unaddressed level of the hierarchy. Fleet is implemented as a persistent kernel runtime with per-chiplet scheduling, allowing workers within a chiplet to cooperatively execute tasks with coordinated cache reuse. On AMD Instinct MI350 with Qwen3-8B, Fleet achieves 1.3-1.5x lower decode latency than vLLM at batch sizes 1-8 through persistent kernel execution and per-chiplet scheduling. At larger batch sizes, cooperative weight tiling increases L2 hit rate (from 12% to 54% at batch size 32 and from 39% to 61% at batch size 64), reducing HBM traffic by up to 37% and delivering 1.27-1.30x speedup over a chiplet-unaware megakernel baseline.