TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning

TL;DR

TensorDIMM introduces a hardware/software co-designed near-memory processing architecture with custom DIMMs to significantly improve memory capacity and bandwidth for deep learning embeddings and tensor operations, boosting performance in recommender systems.

Contribution

The paper presents a novel integrated hardware/software design with custom DIMMs and near-data processing cores to enhance memory efficiency for deep learning embeddings.

Findings

Achieves 6.2-17.6x performance improvement on recommender systems

Enables scalable memory bandwidth and capacity expansion for GPUs

Addresses memory challenges in embedding layers for deep learning

Abstract

Recent studies from several hyperscalars pinpoint to embedding layers as the most memory-intensive deep learning (DL) algorithm being deployed in today's datacenters. This paper addresses the memory capacity and bandwidth challenges of embedding layers and the associated tensor operations. We present our vertically integrated hardware/software co-design, which includes a custom DIMM module enhanced with near-data processing cores tailored for DL tensor operations. These custom DIMMs are populated inside a GPU-centric system interconnect as a remote memory pool, allowing GPUs to utilize for scalable memory bandwidth and capacity expansion. A prototype implementation of our proposal on real DL systems shows an average 6.2-17.6x performance improvement on state-of-the-art recommender systems.