SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention

TL;DR

SparseBalance is a co-designed algorithm-system framework that enhances long-context training efficiency and accuracy by dynamically balancing sparsity and sequence heterogeneity in sparse attention models.

Contribution

It introduces workload-aware dynamic sparsity tuning and a sparsity-aware batching strategy to jointly optimize model accuracy and system efficiency.

Findings

Achieves up to 1.33× speedup in training.

Improves long-context capability by 0.46% on LongBench.

Effectively balances heterogeneity issues in sparse attention training.

Abstract

While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both \textit{1)} sequence length and \textit{2)} sparsity sensitivity, leading to a severe imbalance problem and sub-optimal model accuracy. Existing algorithms and training frameworks typically focus on single issue, failing to systematically co-optimize these two problems. Therefore, we propose SparseBalance, a novel algorithm-system co-design framework, which exploits the sparsity and sequence heterogeneity to optimize model accuracy and system efficiency jointly. First, we propose workload-aware dynamic sparsity tuning, which employs a bidirectional sparsity adjustment to eliminate stragglers and exploit inherent bubbles for free accuracy. Second, we propose a sparsity-aware batching strategy to achieve coarse-grained balance, which complements dynamic sparsity tuning. Experimental results demonstrate that SparseBalance achieves up to a 1.33$\times$ end-to-end speedup while still improving the long-context capability by 0.46\% on the LongBench benchmark.