PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems

TL;DR

This paper introduces PICASSO, a GPU-centric framework that significantly improves training throughput and hardware utilization for wide-and-deep recommender systems, addressing underutilization issues in traditional GPU training.

Contribution

PICASSO systematically analyzes training bottlenecks and proposes novel packing, interleaving, and caching optimizations to enhance GPU utilization for recommendation models.

Findings

Up to 6x throughput improvement over SOTA baselines.

Increases hardware utilization by an order of magnitude.

Reduces daily training walltime by 7 hours on average.

Abstract

The development of personalized recommendation has significantly improved the accuracy of information matching and the revenue of e-commerce platforms. Recently, it has 2 trends: 1) recommender systems must be trained timely to cope with ever-growing new products and ever-changing user interests from online marketing and social network; 2) SOTA recommendation models introduce DNN modules to improve prediction accuracy. Traditional CPU-based recommender systems cannot meet these two trends, and GPU- centric training has become a trending approach. However, we observe that GPU devices in training recommender systems are underutilized, and they cannot attain an expected throughput improvement as what it has achieved in CV and NLP areas. This issue can be explained by two characteristics of these recommendation models: First, they contain up to a thousand input feature fields, introducing fragmentary and memory-intensive operations; Second, the multiple constituent feature interaction submodules introduce substantial small-sized compute kernels. To remove this roadblock to the development of recommender systems, we propose a novel framework named PICASSO to accelerate the training of recommendation models on commodity hardware. Specifically, we conduct a systematic analysis to reveal the bottlenecks encountered in training recommendation models. We leverage the model structure and data distribution to unleash the potential of hardware through our packing, interleaving, and caching optimization. Experiments show that PICASSO increases the hardware utilization by an order of magnitude on the basis of SOTA baselines and brings up to 6x throughput improvement for a variety of industrial recommendation models. Using the same hardware budget in production, PICASSO on average shortens the walltime of daily training tasks by 7 hours, significantly reducing the delay of continuous delivery.