LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

TL;DR

LLaVA-FA introduces a Fourier-based joint low-rank and quantization approach for compressing large multimodal models, significantly reducing computational costs while maintaining high performance.

Contribution

It proposes a novel Fourier domain approximation method combined with PolarQuant and diagonal calibration to improve compression of multimodal models.

Findings

Outperforms existing models on multiple benchmarks.

Achieves high compression with minimal accuracy loss.

Reduces computational and memory costs significantly.

Abstract

Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.