CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure

TL;DR

CCL-Bench is a trace-based benchmark for LLM infrastructure that provides detailed evidence and metrics to better understand performance variations across hardware and software configurations.

Contribution

It introduces a comprehensive, reusable trace-based benchmark with an extensible toolkit for analyzing LLM infrastructure performance.

Findings

Higher compute-communication overlap can hide inefficiencies.

Doubling TPU interconnect bandwidth significantly improves step time.

Framework tuning can cause up to 3x performance differences on identical hardware.

Abstract

Evaluative claims about LLM infrastructure -- ``workload X is fastest on hardware Y with software Z'' -- depend on a complex configuration space spanning hardware accelerators, interconnect bandwidth, software frameworks, parallelism plans, and communication libraries. Current infrastructure evaluation benchmarks publish a small set of end-to-end numbers that do not explain why one configuration outperforms another. We present CCL-Bench, a trace-based benchmark that addresses the limitations of existing benchmarks by recording reusable evidence for every ML workload. Each contributed data point in CCL-Bench packages an execution trace, a YAML workload card, and the launch scripts. We have developed a community-extensible toolkit to compute fine-grained compute, memory, and communication efficiency metrics from this evidence. Using CCL-Bench, we surface three claims that summary-statistic benchmarks cannot support: (i) higher compute-communication overlap can coincide with longer training step time and reveal inefficient parallelization choices, (ii) doubling TPU interconnect bandwidth yields a much higher end-to-end improvement in step time than doubling GPU interconnect bandwidth on small and medium workloads, and (iii) the best-tuned configuration on one training framework can run up to 3$\times$ slower than the best-tuned configuration on a peer framework on identical hardware.