Taming the Tail: NoI Topology Synthesis for Mixed DL Workloads on Chiplet-Based Accelerators

TL;DR

This paper introduces a novel topology synthesis method for chiplet-based accelerators that reduces tail latency and contention in heterogeneous systems with memory-driven workloads, ensuring SLA compliance.

Contribution

It proposes an InterferenceScore metric and formulates NoI topology synthesis as a multi-objective optimization problem solved by PARL, a reinforcement learning-based generator.

Findings

PARL reduces worst-case slowdown to 1.2x

Topologies meet SLAs and reduce memory contention

Maintains competitive throughput with link-rich meshes

Abstract

Heterogeneous chiplet-based systems improve scaling by disag-gregating CPUs/GPUs and emerging technologies (HBM/DRAM).However this on-package disaggregation introduces a latency inNetwork-on-Interposer(NoI). We observe that in modern large-modelinference, parameters and activations routinely move backand forth from HBM/DRAM, injecting large, bursty flows into theinterposer. These memory-driven transfers inflate tail latency andviolate Service Level Agreements (SLAs) across k-ary n-cube base-line NoI topologies. To address this gap we introduce an InterferenceScore (IS) that quantifies worst-case slowdown under contention.We then formulate NoI synthesis as a multi-objective optimization(MOO) problem. We develop PARL (Partition-Aware ReinforcementLearner), a topology generator that balances throughput, latency,and power. PARL-generated topologies reduce contention at the memory cut, meet SLAs, and cut worst-case slowdown to 1.2 times while maintaining competitive mean throughput relative to link-rich meshes. Overall, this reframes NoI design for heterogeneouschiplet accelerators with workload-aware objectives.