Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

TL;DR

This paper introduces Disaggregated Multi-Tower (DMT), a novel topology-aware modeling technique for recommendation systems that improves training efficiency and scalability by exploiting data center locality and hierarchical feature interactions.

Contribution

The paper proposes DMT, combining SPTT, TM, and TP to address inefficiencies in flat architectures, enabling faster training without accuracy loss at large scale.

Findings

Up to 1.9x speedup over state-of-the-art baselines

Maintains accuracy across multiple hardware generations

Effective at large data center scales

Abstract

We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.