Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism

TL;DR

This paper introduces Dynamic Communication Thresholding (DCT), a compression framework for hybrid parallelism in training large recommendation models, significantly reducing communication overhead and improving training efficiency without performance loss.

Contribution

The paper presents a novel DCT framework that efficiently compresses communication in hybrid parallelism, enabling scalable training of large recommendation models with minimal overhead.

Findings

DCT reduces communication by over 100x in data parallelism.

DCT reduces communication by over 20x in model parallelism.

Deployment of DCT improved training time by 37% in production without performance loss.

Abstract

In this paper, we consider hybrid parallelism -- a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP) -- to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least $100\times$ and $20\times$ during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37\%, without any loss in performance.