BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

TL;DR

The paper introduces BLAST, a flexible structured matrix that efficiently compresses large neural networks, reducing computational complexity and maintaining performance across language and vision models.

Contribution

BLAST is a novel adaptive structured matrix that learns and leverages various structures in weight matrices, enabling significant compression and efficiency improvements in large models.

Findings

Reduces complexity by 70% in ViT and 40% in GPT-2.

Achieves 2x compression in large models like Llama-7B and DiT-XL.

Maintains low performance degradation compared to other structured matrices.

Abstract

Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.