MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression

TL;DR

MLoRQ is a novel method that combines low-rank approximation and mixed-precision quantization to efficiently compress transformer models, achieving state-of-the-art performance improvements on vision tasks.

Contribution

Introduces MLoRQ, a two-stage optimization technique integrating low-rank and quantization methods for transformer compression under memory constraints.

Findings

Up to 15% performance improvement on vision transformers.

Compatible with most existing quantization algorithms.

Effective across image classification, object detection, and segmentation.

Abstract

Deploying transformer-based neural networks on resource-constrained edge devices presents a significant challenge. This challenge is often addressed through various techniques, such as low-rank approximation and mixed-precision quantization. In this work, we introduce Mixed Low-Rank and Quantization (MLoRQ), a novel method that integrates both techniques. MLoRQ employs a two-stage optimization process to determine optimal bit-width and rank assignments for each layer, adhering to predefined memory constraints. This process includes: (i) an intra-layer optimization that identifies potentially optimal compression solutions out of all low-rank and quantization combinations; (ii) an inter-layer optimization that assigns bit-width precision and rank to each layer while ensuring the memory constraint is met. An optional final step applies a sequential optimization process using a modified adaptive rounding technique to mitigate compression-induced errors in joint low-rank approximation and quantization. The method is compatible and can be seamlessly integrated with most existing quantization algorithms. MLoRQ shows state-of-the-art results with up to 15\% performance improvement, evaluated on Vision Transformers for image classification, object detection, and instance segmentation tasks.