FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

TL;DR

FlexQuant is an innovative framework that enables elastic, memory-efficient hosting of large language models on edge devices by generating a family of quantized models with fine-grained control and reduced storage requirements.

Contribution

It introduces a novel elasticity framework that improves transition granularity and storage efficiency for LLM deployment on edge devices, compatible with most quantization methods.

Findings

Achieves 15x granularity improvement over state-of-the-art methods.

Reduces storage requirements by 10x compared to existing solutions.

Provides flexible trade-offs between performance and storage under various constraints.

Abstract

Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions suffer from either poor transition granularity or high storage costs. We propose FlexQuant, a novel elasticity framework that generates an ensemble of quantized models, providing an elastic hosting solution with 15x granularity improvement and 10x storage reduction compared to SoTA methods. FlexQuant works with most quantization methods and creates a family of trade-off options under various storage limits through our pruning method. It brings great performance and flexibility to the edge deployment of LLMs.