Camel: Energy-Aware LLM Inference on Resource-Constrained Devices

TL;DR

This paper introduces Camel, an energy-aware framework for LLM inference on resource-constrained edge devices, optimizing GPU settings to reduce energy-delay product while maintaining low latency.

Contribution

Camel is the first framework to optimize GPU frequency and batch size for energy-efficient LLM inference on edge devices, balancing latency and energy consumption.

Findings

Reduces energy delay product by up to 29.9%

Achieves better energy-latency trade-off compared to default settings

Validated on NVIDIA Jetson AGX Orin platform

Abstract

Most Large Language Models (LLMs) are currently deployed in the cloud, with users relying on internet connectivity for access. However, this paradigm faces challenges such as network latency, privacy concerns, and bandwidth limits. Thus, deploying LLMs on edge devices has become an important research focus. In edge inference, request latency is critical as high latency can impair real-time tasks. At the same time, edge devices usually have limited battery capacity, making energy consumption another major concern. Balancing energy consumption and inference latency is essential. To address this, we propose an LLM inference energy management framework that optimizes GPU frequency and batch size to balance latency and energy consumption. By effectively managing the exploration-exploitation dilemma in configuration search, the framework finds the optimal settings. The framework was implemented on the NVIDIA Jetson AGX Orin platform, and a series of experimental validations were conducted. Results demonstrate that, compared to the default configuration, our framework reduces energy delay product (EDP) by 12.4%-29.9%, achieving a better balance between energy consumption and latency.