AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment

TL;DR

AmoebaLLM is a framework that enables instant extraction of optimized, shape-adaptive LLM subnets post fine-tuning, facilitating rapid deployment across diverse platforms with tailored efficiency and accuracy.

Contribution

It introduces a novel method for constructing arbitrary-shape LLM subnets that are immediately deployable after one-time fine-tuning, addressing deployment efficiency challenges.

Findings

Achieves state-of-the-art accuracy-efficiency trade-offs.

Enables rapid, on-demand LLM subnet deployment.

Demonstrates superior adaptability across platforms.

Abstract

Motivated by the transformative capabilities of large language models (LLMs) across various natural language tasks, there has been a growing demand to deploy these models effectively across diverse real-world applications and platforms. However, the challenge of efficiently deploying LLMs has become increasingly pronounced due to the varying application-specific performance requirements and the rapid evolution of computational platforms, which feature diverse resource constraints and deployment flows. These varying requirements necessitate LLMs that can adapt their structures (depth and width) for optimal efficiency across different platforms and application specifications. To address this critical gap, we propose AmoebaLLM, a novel framework designed to enable the instant derivation of LLM subnets of arbitrary shapes, which achieve the accuracy-efficiency frontier and can be extracted immediately after a one-time fine-tuning. In this way, AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications. Specifically, AmoebaLLM integrates three innovative components: (1) a knowledge-preserving subnet selection strategy that features a dynamic-programming approach for depth shrinking and an importance-driven method for width shrinking; (2) a shape-aware mixture of LoRAs to mitigate gradient conflicts among subnets during fine-tuning; and (3) an in-place distillation scheme with loss-magnitude balancing as the fine-tuning objective. Extensive experiments validate that AmoebaLLM not only sets new standards in LLM adaptability but also successfully delivers subnets that achieve state-of-the-art trade-offs between accuracy and efficiency.