Dissecting Embedding Bag Performance in DLRM Inference

TL;DR

This paper evaluates the performance of Embedding Bag kernels in large-scale DLRMs on H100 GPUs, focusing on the impact of model partitioning across multiple GPUs and the associated communication overhead.

Contribution

It provides a detailed analysis of Embedding Bag performance in distributed DLRMs, highlighting the effects of partitioning and communication overhead on inference efficiency.

Findings

Partitioning large embedding tables across GPUs causes significant performance slowdown.

Communication overhead varies with batch size, number of tables, and embedding dimensions.

Using NCCL and NVSHMEM libraries helps measure and understand performance bottlenecks.

Abstract

As the size of DLRMs gets larger, the models must be partitioned across multiple GPUs or nodes of GPUs due to the size limitation of total HBM memory that can be packaged in a GPU. This partitioning adds communication and synchronization overhead of sending and receiving data across GPUs. We use the NCCL and NVSHMEM libraries to measure the performance of an Embedding Bag kernel implemented on H100 GPUs. We compare its performance across diOerent batch sizes, number of tables, table sizes, pooling factors, and embedding dimensions. For a large embedding table that spans multiple GPUs, we project the performance slowdown from distributing an embedding table across multiple GPUs.