Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers

TL;DR

This paper introduces SARDFQ, a novel data-free quantization method for Vision Transformers that enhances semantic quality of synthetic images through alignment and reinforcement, significantly improving quantization accuracy.

Contribution

SARDFQ is the first to combine semantic alignment and reinforcement techniques specifically for data-free quantization of Vision Transformers, addressing key semantic issues.

Findings

Improves top-1 accuracy on ImageNet by 15.52% for W4A4 ViT-B.

Effectively reduces semantic distortion and inadequacy in synthetic images.

Outperforms existing data-free quantization methods significantly.

Abstract

Data-free quantization (DFQ) enables model quantization without accessing real data, addressing concerns regarding data security and privacy. With the growing adoption of Vision Transformers (ViTs), DFQ for ViTs has garnered significant attention. However, existing DFQ methods exhibit two limitations: (1) semantic distortion, where the semantics of synthetic images deviate substantially from those of real images, and (2) semantic inadequacy, where synthetic images contain extensive regions with limited content and oversimplified textures, leading to suboptimal quantization performance. To address these limitations, we propose SARDFQ, a novel Semantics Alignment and Reinforcement Data-Free Quantization method for ViTs. To address semantic distortion, SARDFQ incorporates Attention Priors Alignment (APA), which optimizes synthetic images to follow randomly generated structure attention priors. To mitigate semantic inadequacy, SARDFQ introduces Multi-Semantic Reinforcement (MSR), leveraging localized patch optimization to enhance semantic richness across synthetic images. Furthermore, SARDFQ employs Soft-Label Learning (SL), wherein multiple semantic targets are adapted to facilitate the learning of multi-semantic images augmented by MSR. Extensive experiments demonstrate the effectiveness of SARDFQ, significantly surpassing existing methods. For example, SARDFQ improves top-1 accuracy on ImageNet by 15.52% for W4A4 ViT-B. The code is at https://github.com/zysxmu/SARDFQ.