Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap

TL;DR

Syncopate is a compiler and runtime that enables automatic fine-grained compute-communication overlap within a single GPU kernel, significantly improving multi-GPU workload performance.

Contribution

It introduces a chunk abstraction and transformations for fine-grained overlap, enabling more efficient multi-GPU communication handling.

Findings

Achieves an average 1.3× speedup on multi-GPU workloads.

Up to 4.7× speedup in certain cases.

Enables reuse of chunk plans from existing compilers or templates.

Abstract

Communication has become a first-order bottleneck in large-cale GPU workloads, and existing distributed compilers address it mainly by overlapping whole compute and communication kernels at the stream level. This coarse granularity incurs extra kernel launches, forces device-wide synchronizations at kernel boundaries, and leaves substantial slack when the slowest tile or kernel stretches the communication tail. We present Syncopate, a compiler and runtime that enables automatic fine-grained overlap inside a single fused kernel. Syncopate introduces a communication chunk abstraction that decouples communication granularity from kernel structure and backend mechanisms, allowing chunk-level plans to be ported from existing distributed compilers, written directly by users, or instantiated from reusable templates. Given a local Triton kernel and a chunk schedule, Syncopate performs transformations to align computation with chunk availability. Implemented as a source-to-source compiler on Triton, Syncopate delivers an average end-to-end speedup of 1.3$\times$ and up to 4.7$\times$ on multi-GPU workloads.