Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters

TL;DR

This paper introduces a variability modeling approach to systematically analyze and optimize inference hyperparameters of large language models, improving energy efficiency and understanding trade-offs.

Contribution

It applies variability management techniques to LLM inference configurations, enabling systematic analysis and predictive modeling of hyperparameter effects.

Findings

Variability modeling manages complexity of LLM inference configurations.

It reveals hyperparameter trade-offs and interactions.

Predictive models can estimate inference behavior from limited data.

Abstract

Large Language Models (LLMs) are being increasingly used across a wide range of tasks. However, their substantial computational demands raise concerns about the energy efficiency and sustainability of both training and inference. Inference, in particular, dominates total compute usage, making its optimization crucial. Recent research has explored optimization techniques and analyzed how configuration choices influence energy consumption. Yet, the vast configuration space of inference servers makes exhaustive empirical evaluation infeasible due to combinatorial explosion. In this paper, we introduce a new perspective on this problem by treating LLMs as configurable systems and applying variability management techniques to systematically analyze inference-time configuration choices. We evaluate our approach on the Hugging Face Transformers library by representing generation hyperparameters and their constraints using a feature-based variability model, sampling representative configurations, measuring their energy consumption, latency, accuracy, and learning predictive models from the collected data. Our results show that variability modeling effectively manages the complexity of LLM inference configurations. It enables systematic analysis of hyperparameters effects and interactions, reveals trade-offs, and supports prediction of inference behavior from a limited number of measurements. Overall, this work opens a new research direction that bridges software engineering and machine learning by leveraging variability modeling for the efficient and sustainable configuration of LLMs.