Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling

TL;DR

LIFE is a hardware-agnostic analytical framework that accurately forecasts LLM inference performance across diverse devices by modeling operator behavior and optimizations using only hardware specifications.

Contribution

The paper introduces LIFE, a modular analytical model that predicts LLM inference performance without extensive benchmarking, accommodating hardware and software optimizations.

Findings

LIFE accurately forecasts inference metrics on various hardware platforms.

The framework accounts for software optimizations like quantization and operator fusion.

Validation shows LIFE's predictions align well with real-world inference performance.

Abstract

Large language models (LLMs) have been increasingly deployed as local agents on personal devices with CPUs, NPUs and integrated GPUs. However, forecasting inference performance on devices with such heterogeneity remains challenging due to the dynamic compute and memory demands. Existing approaches rely on GPU benchmarking or machine learning-based latency predictors, which are often hardware-specific and lack generalizability. To this end, we introduce LIFE, a lightweight and modular analytical framework that is comprised of modular analytical model of operators, configurable to characterize LLM inference workloads in a hardware and dataset-agnostic manner. LIFE characterizes the influence of software and model optimizations, such as quantization, KV cache compression, LoRA adapters, chunked prefill, different attentions, and operator fusion, on performance metrics such as time-to-first-token (TTFT), time-per-output-token (TPOT) and tokens-per-second (TPS). LIFE enables performance forecasting using only hardware specifications, such as TOPS and memory bandwidth, without requiring extensive dataset benchmarking. We validate LIFE's forecasting with inference on AMD Ryzen CPUs, NPUs, iGPUs and NVIDIA V100 GPUs, with Llama2-7B variants, demonstrating the utility of LIFE in forecasting LLM performance through lens of system efficiency to enable efficient LLM deployment across different hardware platforms.