EvoPress: Accurate Dynamic Model Compression via Evolutionary Search

TL;DR

EvoPress introduces an evolutionary approach to optimize dynamic, non-uniform compression of large language models, significantly reducing computational costs while maintaining accuracy across various models and compression techniques.

Contribution

It presents a novel evolutionary framework for dynamic LLM compression that outperforms existing methods and generalizes across multiple models and compression strategies.

Findings

Achieved state-of-the-art results on Llama, Mistral, and Phi models.

Set new benchmarks for structural pruning, sparsity, and quantization.

Demonstrated the effectiveness of evolutionary search in model compression.

Abstract

The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pruning. A new frontier in this area is given by dynamic, non-uniform compression methods, which adjust the compression levels (e.g., sparsity) per-block or even per-layer in order to minimize accuracy loss, while guaranteeing a global compression threshold. Yet, current methods rely on estimating the importance of a given layer, implicitly assuming that layers contribute independently to the overall compression error. We begin from the motivating observation that this independence assumption does not generally hold for LLM compression: pruning a model further may even significantly recover performance. To address this, we propose EvoPress, a novel evolutionary framework for dynamic LLM compression. By formulating dynamic compression as a general optimization problem, EvoPress identifies optimal compression profiles in a highly efficient manner, and generalizes across diverse models and compression techniques. Via EvoPress, we achieve state-of-the-art performance for dynamic compression of Llama, Mistral, and Phi models, setting new benchmarks for structural pruning (block/layer dropping), unstructured sparsity, and quantization with dynamic bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress}.