LatentLLM: Attention-Aware Joint Tensor Compression

TL;DR

LatentLLM introduces an attention-aware tensor decomposition framework that compresses large language and multi-modal models into efficient, lower-dimensional structures, improving accuracy over existing methods on various benchmarks.

Contribution

The paper presents a novel global attention-aware joint tensor decomposition method for model compression, enhancing accuracy in reduced-dimension LLMs and LMMs.

Findings

Significant accuracy improvements over existing compression methods.

Effective reduction of model size for multi-modal reasoning tasks.

Demonstrated benefits on multiple benchmark datasets.

Abstract

Modern foundation models such as large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources. We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure. Our method extends a local activation-aware tensor decomposition to a global attention-aware joint tensor de-composition. Our framework can significantly improve the model accuracy over the existing model compression methods when reducing the latent dimension to realize computationally/memory-efficient LLMs/LLMs. We show the benefit on several benchmark including multi-modal reasoning tasks.