Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and NVIDIA Data Center GPUs

TL;DR

This paper benchmarks the Qualcomm Cloud AI 100 Ultra against NVIDIA A100 GPUs for large language model inference, highlighting energy efficiency and hardware scalability advantages in HPC environments.

Contribution

It provides a comprehensive comparison of QAic and NVIDIA GPUs for LLM inference, demonstrating QAic's superior energy efficiency and scalability for large models.

Findings

QAic achieves competitive energy efficiency with model-specific advantages.

Large models can run on fewer QAic cards, reducing power consumption.

QAic enables resource-efficient HPC deployments for LLM inference.

Abstract

This study presents a benchmarking analysis of the Qualcomm Cloud AI 100 Ultra (QAic) accelerator for large language model (LLM) inference, evaluating its energy efficiency (throughput per watt), performance, and hardware scalability against NVIDIA A100 GPUs (in 4x and 8x configurations) within the National Research Platform (NRP) ecosystem. A total of 12 open-source LLMs, ranging from 124 million to 70 billion parameters, are served using the vLLM framework. Our analysis reveals that QAic achieves competitive energy efficiency with advantages on specific models while enabling more granular hardware allocation: some 70B models operate on as few as 1 QAic card versus 8 A100 GPUs required, with 20x lower power consumption (148W vs 2,983W). For smaller models, single QAic devices achieve up to 35x lower power consumption compared to our 4-GPU A100 configuration (36W vs 1,246W). The findings offer insights into the potential of the Qualcomm Cloud AI 100 Ultra for energy-constrained and resource-efficient HPC deployments within the National Research Platform (NRP).