DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuning

TL;DR

This paper introduces DFQ-ViT, a data-free quantization method for Vision Transformers that synthesizes high-quality data and aligns activations, achieving near data-driven performance without fine-tuning.

Contribution

It proposes a novel pipeline for data-free quantization of ViTs, including difficulty-based sample synthesis and activation correction, improving performance without fine-tuning.

Findings

DFQ-ViT outperforms existing data-free methods.

Achieves performance comparable to real-data quantization.

Reduces computational overhead and deployment barriers.

Abstract

Data-Free Quantization (DFQ) enables the quantization of Vision Transformers (ViTs) without requiring access to data, allowing for the deployment of ViTs on devices with limited resources. In DFQ, the quantization model must be calibrated using synthetic samples, making the quality of these synthetic samples crucial. Existing methods fail to fully capture and balance the global and local features within the samples, resulting in limited synthetic data quality. Moreover, we have found that during inference, there is a significant difference in the distributions of intermediate layer activations between the quantized and full-precision models. These issues lead to a severe performance degradation of the quantized model. To address these problems, we propose a pipeline for Data-Free Quantization for Vision Transformers (DFQ-ViT). Specifically, we synthesize samples in order of increasing difficulty, effectively enhancing the quality of synthetic data. During the calibration and inference stage, we introduce the activation correction matrix for the quantized model to align the intermediate layer activations with those of the full-precision model. Extensive experiments demonstrate that DFQ-ViT achieves remarkable superiority over existing DFQ methods and its performance is on par with models quantized through real data. For example, the performance of DeiT-T with 3-bit weights quantization is 4.29% higher than the state-of-the-art. Our method eliminates the need for fine-tuning, which not only reduces computational overhead but also lowers the deployment barriers for edge devices. This characteristic aligns with the principles of Green Learning by improving energy efficiency and facilitating real-world applications in resource-constrained environments.