Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models

TL;DR

This paper introduces ESC, an evolution strategy-based calibration method that optimizes activation scaling for low-bit speech model quantization, achieving near-lossless INT4 performance and minimal accuracy loss.

Contribution

The paper presents ESC, a novel calibration approach tailored for speech models that effectively handles large activation ranges and enables ultra-low-bit quantization with minimal accuracy loss.

Findings

ESC achieves near-lossless INT4 quantization across speech tasks.

Combining ESC with PTQ reduces accuracy degradation to 1%.

ESC maintains full INT8 performance with standard calibration techniques.

Abstract

Quantization has become essential for the efficient deployment of speech processing systems. Although widely studied, most existing quantization methods were developed for vision and NLP architectures, while the specific challenges of audio signals remain largely overlooked. In particular, we show that audio activations can exhibit large calibration ranges, leading to significant information loss when standard calibration techniques are applied. To address this, we propose ESC, an Evolution Strategy-based Calibration method that formulates activation scaling as an optimization problem and solves it using a two-step local-global scheme driven by an evolution strategy. ESC enables unaltered performance under full INT8 quantization and is the first calibration method to achieve near-lossless performance for full INT4 quantization across multiple speech tasks. Integrating ESC with PTQ methods further reduces performance loss, achieving a 1% relative accuracy degradation on the AST model.