CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs

TL;DR

CLAMP-ViT introduces a data-free post-training quantization method for vision transformers that uses contrastive learning to generate meaningful data and optimize quantization parameters, significantly improving accuracy across multiple vision tasks.

Contribution

It proposes a novel two-stage contrastive learning framework for data-free quantization of ViTs, enhancing data generation and quantization parameter selection.

Findings

Up to 3% accuracy improvement in classification

0.6 mAP enhancement in object detection

1.5 mIoU increase in segmentation

Abstract

We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs). We identify the limitations of recent techniques, notably their inability to leverage meaningful inter-patch relationships, leading to the generation of simplistic and semantically vague data, impacting quantization accuracy. CLAMP-ViT employs a two-stage approach, cyclically adapting between data generation and model quantization. Specifically, we incorporate a patch-level contrastive learning scheme to generate richer, semantically meaningful data. Furthermore, we leverage contrastive learning in layer-wise evolutionary search for fixed- and mixed-precision quantization to identify optimal quantization parameters while mitigating the effects of a non-smooth loss landscape. Extensive evaluations across various vision tasks demonstrate the superiority of CLAMP-ViT, with performance improvements of up to 3% in top-1 accuracy for classification, 0.6 mAP for object detection, and 1.5 mIoU for segmentation at similar or better compression ratio over existing alternatives. Code is available at https://github.com/georgia-tech-synergy-lab/CLAMP-ViT.git