RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis

TL;DR

This paper introduces RooflineBench, a benchmarking framework based on the Roofline model, to evaluate and compare the performance and efficiency of on-device Large Language Models across heterogeneous hardware platforms.

Contribution

It proposes a systematic Roofline-based framework and a new metric, Relative Inference Potential, for analyzing LLM performance on resource-constrained hardware.

Findings

Performance and operational intensity vary with sequence length.

Model depth regression impacts operational intensity.

Structural refinements like Multi-head Latent Attention improve inference efficiency.

Abstract

The transition toward localized intelligence through Small Language Models (SLMs) has intensified the need for rigorous performance characterization on resource-constrained edge hardware. However, objectively measuring the theoretical performance ceilings of diverse architectures across heterogeneous platforms remains a formidable challenge. In this work, we propose a systematic framework based on the Roofline model that unifies architectural primitives and hardware constraints through the lens of operational intensity (OI). By defining an inference-potential region, we introduce the Relative Inference Potential as a novel metric to compare efficiency differences between Large Language Models (LLMs) on the same hardware substrate. Extensive empirical analysis across diverse compute tiers reveals that variations in performance and OI are significantly influenced by sequence length. We further identify a critical regression in OI as model depth increases. Additionally, our findings highlight an efficiency trap induced by hardware heterogeneity and demonstrate how structural refinements, such as Multi-head Latent Attention (M LA), can effectively unlock latent inference potential across various hardware substrates. These insights provide actionable directions for hardware-software co-design to align neural structures with physical constraints in on-device intelligence. The released code is available in the Appendix C.