QuantFace: Efficient Quantization for Face Restoration

TL;DR

QuantFace introduces a low-bit quantization framework for face restoration models, significantly reducing computational costs while maintaining high performance through innovative data distribution analysis, adaptive bit-width allocation, and joint optimization techniques.

Contribution

It presents a novel low-bit quantization method with adaptive bit-width allocation and a joint optimization strategy for face restoration models, improving efficiency and effectiveness.

Findings

QuantFace performs well at 4-bit and 6-bit quantization levels.

It outperforms recent low-bit quantization methods in face restoration.

The approach reduces computational costs significantly.

Abstract

Diffusion models have been achieving remarkable performance in face restoration. However, the heavy computations hamper the widespread adoption of these models. In this work, we propose QuantFace, a novel low-bit quantization framework for face restoration models, where the full-precision (i.e., 32-bit) weights and activations are quantized to 4~6-bit. We first analyze the data distribution within activations and find that it is highly variant. To preserve the original data information, we employ rotation-scaling channel balancing. Furthermore, we propose Quantization-Distillation Low-Rank Adaptation (QD-LoRA), which jointly optimizes for quantization and distillation performance. Finally, we propose an adaptive bit-width allocation strategy. We formulate such a strategy as an integer programming problem that combines quantization error and perceptual metrics to find a satisfactory resource allocation. Extensive experiments on the synthetic and real-world datasets demonstrate the effectiveness of QuantFace under 6-bit and 4-bit. QuantFace achieves significant advantages over recent leading low-bit quantization methods for face restoration. The code is available at https://github.com/jiatongli2024/QuantFace.