A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models

TL;DR

This paper introduces HugeCTR Hierarchical Parameter Server (HPS), a GPU-optimized distributed framework that significantly reduces inference latency and increases throughput for large-scale recommendation models using a hierarchical storage and caching system.

Contribution

The paper presents a novel GPU-specialized inference framework with hierarchical storage and caching, enabling low-latency, high-throughput recommendation inference at large scale.

Findings

Achieves 5-62x speedup over CPU baselines.

Reduces end-to-end inference latency significantly.

Enables multi-GPU deployment for higher QPS.

Abstract

Recommendation systems are of crucial importance for a variety of modern apps and web services, such as news feeds, social networks, e-commerce, search, etc. To achieve peak prediction accuracy, modern recommendation models combine deep learning with terabyte-scale embedding tables to obtain a fine-grained representation of the underlying data. Traditional inference serving architectures require deploying the whole model to standalone servers, which is infeasible at such massive scale. In this paper, we provide insights into the intriguing and challenging inference domain of online recommendation systems. We propose the HugeCTR Hierarchical Parameter Server (HPS), an industry-leading distributed recommendation inference framework, that combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. Among other things, HPS features (1) a redundant hierarchical storage system, (2) a novel high-bandwidth cache to accelerate parallel embedding lookup on NVIDIA GPUs, (3) online training support and (4) light-weight APIs for easy integration into existing large-scale recommendation workflows. To demonstrate its capabilities, we conduct extensive studies using both synthetically engineered and public datasets. We show that our HPS can dramatically reduce end-to-end inference latency, achieving 5~62x speedup (depending on the batch size) over CPU baseline implementations for popular recommendation models. Through multi-GPU concurrent deployment, the HPS can also greatly increase the inference QPS.