FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression

TL;DR

FLAT-LLM introduces a training-free, fine-grained low-rank transformation technique for compressing large language models, significantly reducing computational demands while maintaining high accuracy and enabling faster inference.

Contribution

It proposes a novel, training-free structural compression method using eigenvector-based low-rank transformations and adaptive rank redistribution for LLMs.

Findings

Outperforms structural pruning in generalization and downstream tasks

Achieves inference speedups over existing decomposition methods

Completes calibration within a few minutes

Abstract

Large Language Models (LLMs) have enabled remarkable progress in natural language processing, yet their high computational and memory demands pose challenges for deployment in resource-constrained environments. Although recent low-rank decomposition methods offer a promising path for structural compression, they often suffer from accuracy degradation, expensive calibration procedures, and result in inefficient model architectures that hinder real-world inference speedups. In this paper, we propose FLAT-LLM, a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. Specifically, we reduce the hidden dimension by transforming the weights using truncated eigenvectors computed via head-wise Principal Component Analysis, and employ a greedy budget redistribution strategy to adaptively allocate ranks across decoders. FLAT-LLM achieves efficient and effective weight compression without recovery fine-tuning, which could complete the calibration within a few minutes. Evaluated across 5 models and 11 datasets, FLAT-LLM outperforms structural pruning baselines in generalization and downstream performance, while delivering inference speedups over decomposition-based methods.