How Much Progress Has There Been in NVIDIA Datacenter GPUs?

TL;DR

This paper analyzes the technical progress of NVIDIA datacenter GPUs from the mid-2000s to present, quantifying improvements in performance, memory, cost, and power, and discusses implications of export controls on future performance gaps.

Contribution

It provides a comprehensive dataset and trend analysis of NVIDIA datacenter GPU features, including performance doubling times and the impact of export regulations.

Findings

FP16 and FP32 operations double approximately every 1.5-1.7 years.

Memory bandwidth and size grow slower, doubling every 3.3-3.5 years.

Export controls could significantly reduce the performance gap in future GPU development.

Abstract

Graphics Processing Units (GPUs) are the state-of-the-art architecture for essential tasks, ranging from rendering 2D/3D graphics to accelerating workloads in supercomputing centers and, of course, Artificial Intelligence (AI). As GPUs continue improving to satisfy ever-increasing performance demands, analyzing past and current progress becomes paramount in determining future constraints on scientific research. This is particularly compelling in the AI domain, where rapid technological advancements and fierce global competition have led the United States to recently implement export control regulations limiting international access to advanced AI chips. For this reason, this paper studies technical progress in NVIDIA datacenter GPUs released from the mid-2000s until today. Specifically, we compile a comprehensive dataset of datacenter NVIDIA GPUs comprising several features, ranging from computational performance to release price. Then, we examine trends in main GPU features and estimate progress indicators for per-memory bandwidth, per-dollar, and per-watt increase rates. Our main results identify doubling times of 1.44 and 1.69 years for FP16 and FP32 operations (without accounting for sparsity benefits), while FP64 doubling times range from 2.06 to 3.79 years. Off-chip memory size and bandwidth grew at slower rates than computing performance, doubling every 3.32 to 3.53 years. The release prices of datacenter GPUs have roughly doubled every 5.1 years, while their power consumption has approximately doubled every 16 years. Finally, we quantify the potential implications of current U.S. export control regulations in terms of the potential performance gaps that would result if implementation were assumed to be complete and successful. We find that recently proposed changes to export controls would shrink the potential performance gap from 23.6x to 3.54x.