MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan

TL;DR

MTGenRec is a scalable distributed system designed for efficient training of generative recommendation models, significantly improving throughput and scalability in industrial settings like Meituan.

Contribution

The paper introduces MTGenRec, a novel system with dynamic hash tables and load balancing techniques for effective GRM training at scale.

Findings

Training throughput increased by 1.6x to 2.4x.

System scales effectively over 100 GPUs.

Deployed in Meituan, improving user engagement metrics.

Abstract

Recommendation is crucial for both user experience and company revenue in Meituan as a leading lifestyle company, and generative recommendation models (GRMs) are shown to produce quality recommendations recently. However, existing systems are limited by insufficient functionality support and inefficient implementations for training GRMs in industrial scenarios. As such, we introduce MTGenRec as an efficient and scalable system for GRM training. Specifically, to handle real-time insertions/deletions of sparse embeddings, MTGenRec employs dynamic hash tables to replace static ones. To improve training efficiency, MTGenRec conducts dynamic sequence balancing to address the computation load imbalances among GPUs and adopts feature ID deduplication alongside automatic table merging to accelerate embedding lookup. Extensive experiments show that MTGenRec improves training throughput by $1.6\times -- 2.4\times$ while achieving good scalability when running over 100 GPUs. MTGenRec has been deployed for many applications in Meituan and is now handling hundreds of millions of requests on a daily basis. On the delivery platform, we observe a 1.22% growth in user order volume and a 1.31% enhancement in online PV_CTR.