HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

TL;DR

HW-GPT-Bench is a hardware-aware benchmark that uses surrogate models to efficiently evaluate and optimize GPT-2 based language models across multiple hardware metrics and devices.

Contribution

It introduces a surrogate-based benchmarking framework for rapid hardware metric estimation of GPT-2 architectures on diverse devices.

Findings

Accurately models latency and energy with calibrated surrogates.

Enables fast simulation of multi-objective optimization trajectories.

Supports evaluation of models up to 1.55B parameters.

Abstract

The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consumption, GPU memory usage, and performance. Identifying optimal model configurations under specific hardware constraints is becoming essential but remains challenging due to the computational load of exhaustive training and evaluation on multiple devices. To address this, we introduce HW-GPT-Bench, a hardware-aware benchmark that utilizes surrogate predictions to approximate various hardware metrics across 13 devices of architectures in the GPT-2 family, with architectures containing up to 1.55B parameters. Our surrogates, via calibrated predictions and reliable uncertainty estimates, faithfully model the heteroscedastic noise inherent in the energy and latency measurements. To estimate perplexity, we employ weight-sharing techniques from Neural Architecture Search (NAS), inheriting pretrained weights from the largest GPT-2 model. Finally, we demonstrate the utility of HW-GPT-Bench by simulating optimization trajectories of various multi-objective optimization algorithms in just a few seconds.