TAPS: Topology-Aware Intra-Operator Parallelism Strategy Searching Algorithm for Deep Neural Networks

TL;DR

TAPS is a topology-aware algorithm that optimizes intra-operator parallelism strategies for deep neural networks by considering network topology, significantly reducing communication costs in multi-node setups.

Contribution

The paper introduces a novel topology-aware cost model for intra-operator parallelism strategy search, improving communication efficiency over existing methods.

Findings

Achieves up to 85% reduction in communication costs

Outperforms latest baseline strategies in experiments

Effectively considers both intra-node and inter-node bandwidths

Abstract

TAPS is a Topology-Aware intra-operator Parallelism strategy Searching algorithm that generates intra-operator parallelism strategies by considering both intra-node and inter-node bandwidth. Most of the existing auto-parallelism works use the communication volume as the communication cost directly when generating strategies, which we prove to be sub-optimal in multi-nodes cases. We design a topology-aware cost model for multi-node intra-operator parallelism strategy searching. Numerical experiments demonstrate that TAPS can generate strategies with up to 85% fewer communication costs, which outperform the latest baselines.