Resource Allocation for Stable LLM Training in Mobile Edge Computing

TL;DR

This paper proposes a resource-efficient collaborative training framework for large language models in mobile edge computing, combining parameter-efficient fine-tuning with optimization techniques to reduce energy, delay, and improve stability.

Contribution

It introduces a novel multi-objective optimization approach integrating PEFT methods for stable LLM training in mobile edge environments.

Findings

Reduces energy consumption during training

Lowers latency in model deployment

Enhances model stability and reliability

Abstract

As mobile devices increasingly become focal points for advanced applications, edge computing presents a viable solution to their inherent computational limitations, particularly in deploying large language models (LLMs). However, despite the advancements in edge computing, significant challenges remain in efficient training and deploying LLMs due to the computational demands and data privacy concerns associated with these models. This paper explores a collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency. Our approach leverages parameter-efficient fine-tuning (PEFT) methods, allowing mobile users to adjust the initial layers of the LLM while edge servers handle the more demanding latter layers. Specifically, we formulate a multi-objective optimization problem to minimize the total energy consumption and delay during training. We also address the common issue of instability in model performance by incorporating stability enhancements into our objective function. Through novel fractional programming technique, we achieve a stationary point for the formulated problem. Simulations demonstrate that our method reduces the energy consumption as well as the latency, and increases the reliability of LLMs across various mobile settings.