An Inquiry into Datacenter TCO for LLM Inference with FP8

TL;DR

This paper analyzes the total cost of ownership for LLM inference in datacenters, emphasizing the impact of low-precision FP8 computation and hardware utilization, providing a framework for comparing AI accelerators.

Contribution

It introduces a generalizable TCO comparison framework for AI accelerators, focusing on FP8 quantization and workload characteristics, with empirical insights into hardware performance.

Findings

Gaudi HPUs outperform others on thin GEMMs in FP8 models

Memory-bound GEMV computations influence TCO more than hardware peak throughput

Empirical workload analysis is crucial for evaluating accelerator performance

Abstract

As large language models (LLMs) continue to scale, the high power consumption of AI accelerators in datacenters presents significant challenges, substantially increasing the total cost of ownership (TCO) for cloud service providers (CSPs) that provide LLM inference. In this work, we analyze the computational characteristics of LLM inference from a TCO perspective and present a generalizable framework to compare AI accelerators across diverse operational requirements. Using this model, we investigate key workload characteristics influencing TCO for AI accelerators from Intel (Gaudi 2 & 3) and NVIDIA (H100 & H200), especially thin GEMM utilization and FP8 quantization. In particular, as FP8 emerges as the baseline precision for next-generation LLMs, understanding how different architectures implement and benefit from low-precision computation is increasingly critical. Throughput on thin GEMMs has a greater impact on TCO than theoretical hardware peak throughput because the memory-bound decode phase is dominated by GEMV-like computations. We find that Gaudi HPUs achieve superior utilization on thin GEMMs compared to their counterparts, especially in FP8-quantized models. Our result underscores the importance of empirical, workload-level analysis in evaluating accelerator performance, rather than relying solely on theoretical hardware specifications. By studying the interaction between power consumption, quantization strategies, and hardware architecture, we provide insights to support informed deployment decisions and guide future accelerator designs aimed at improving the TCO of LLM inference workloads.