AE-LLM: Adaptive Efficiency Optimization for Large Language Models

TL;DR

AE-LLM is a unified framework that automatically optimizes efficiency techniques for large language models, balancing accuracy, latency, memory, and energy to improve deployment performance across diverse scenarios.

Contribution

It introduces a multi-objective optimization approach that dynamically selects and combines efficiency techniques tailored to specific deployment needs, outperforming static configurations.

Findings

Achieves 2.8x efficiency improvement on average across models and tasks.

Maintains accuracy within 1.2% of baseline despite efficiency gains.

Generalizes effectively to vision-language models.

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse applications, yet their deployment remains challenging due to substantial computational costs, memory requirements, and energy consumption. Recent empirical studies have demonstrated that no single efficiency technique is universally optimal; instead, the effectiveness of methods such as efficient attention mechanisms, mixture-of-experts (MoE), parameter-efficient fine-tuning, and quantization varies significantly depending on task characteristics, resource constraints, and model scales. Building upon these insights, we propose AE-LLM, a unified framework that automatically selects and combines optimal efficiency techniques tailored to specific deployment scenarios. Our approach introduces a multi-objective optimization framework that jointly considers accuracy, latency, memory footprint, and energy consumption, while accounting for hardware constraints and task requirements. We develop an efficient search algorithm that explores the combinatorial space of efficiency techniques across architecture, fine-tuning, and inference stages, identifying Pareto-optimal configurations. Extensive experiments across 15 models (0.5B-70B parameters) and 10 diverse tasks demonstrate that AE-LLM achieves an average of $2.8\times$ improvement in efficiency metrics while maintaining competitive accuracy (within 1.2\% of baseline), compared to static efficiency configurations. Furthermore, our framework generalizes effectively to vision-language models, achieving similar efficiency gains. Our contributions provide practitioners with an automated tool for navigating the complex trade-off landscape of LLM efficiency optimization.