SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning
Minjun Kim, Jongjin Kim, and U Kang
TL;DR
SynQ introduces a synthesis-aware fine-tuning framework for zero-shot quantization that effectively reduces noise, aligns class activation maps, and uses soft labels to enhance accuracy without data access.
Contribution
The paper presents SynQ, a novel zero-shot quantization method that overcomes noise, off-target predictions, and label errors through filtering, activation alignment, and soft label guidance.
Findings
SynQ achieves state-of-the-art accuracy among ZSQ methods.
Filtering reduces noise in synthetic data effectively.
Alignment of class activation maps improves quantization quality.
Abstract
How can we accurately quantize a pre-trained model without any data? Quantization algorithms are widely used for deploying neural networks on resource-constrained edge devices. Zero-shot Quantization (ZSQ) addresses the crucial and practical scenario where training data are inaccessible for privacy or security reasons. However, three significant challenges hinder the performance of existing ZSQ methods: 1) noise in the synthetic dataset, 2) predictions based on off-target patterns, and the 3) misguidance by erroneous hard labels. In this paper, we propose SynQ (Synthesis-aware Fine-tuning for Zero-shot Quantization), a carefully designed ZSQ framework to overcome the limitations of existing methods. SynQ minimizes the noise from the generated samples by exploiting a low-pass filter. Then, SynQ trains the quantized model to improve accuracy by aligning its class activation map with the…
Peer Reviews
Decision·ICLR 2025 Poster
1. SYNQ offers a unique solution to the problem of quantizing models without access to training data, which is a significant contribution to deploying neural networks on edge devices. 2. Addressing Key Challenges: The paper clearly identifies and addresses three major challenges in ZSQ, providing a comprehensive approach to improving the accuracy of quantized models. 3. Empirical Validation: Extensive experiments demonstrate SYNQ's effectiveness, showing improvements in classification accuracy o
1. While the paper focuses on image classification, it's unclear how SYNQ would perform in other tasks such as object detection or segmentation. 2. The paper could provide more details on the computational overhead introduced by SYNQ, especially the impact of the low-pass filter and CAM alignment. 3. The paper could benefit from a deeper analysis of SYNQ's robustness to different types and levels of noise in synthetic datasets.
1. The observations regarding the three limitations of ZSQ are interesting, and the proposed method appears feasible. 2. The performance is validated through a variety of experiments. Specifically, experiments were conducted to verify the performance of SYNQ by comparing it with various ZDQ baselines on not only CNN-based models but also ViT-based models. 3. The detailed analyses of the three components of SYNQ enhance the persuasiveness of the methodology. 4. This paper is well-written and easy
1. Although the observations presented in the paper are interesting, most of the experimental evidence provided was gathered under limited conditions. For instance, in Figure 5, experiments were shown only for TexQ among various baseline models, and the analysis for CIFAR-10 and CIFAR-100 used as benchmarks in Table 1 was omitted. 2. In Figure 2, the heat map is shown only one sample image. For these reasons, it is difficult to be certain whether the presented observations are phenomena that ca
- The paper tackles the limitations of previous works well. - The paper tries to denoise synthesized images with a loss-pass filter. This idea is a good point that highlights the importance of classical techniques and theories in the recent AI era. - The paper identifies the off-target prediction problem that occurs only in the data-free quantization scenario. It is a good suggestion that analyzes the performance of a quantized model with grad GAM and uses it as a loss function. - The paper exec
- The paper refers to the limitations of previous works too much. The same content is repeated over 3 times. - The investigation and analysis of prior works is insufficient. - The paper notes that using hard labels can be harmful to the performance of quantized models, pointing out that previous works used both hard labels (CE loss) and soft labels (KL divergence loss). It can be a novelty that determines the usage of CE loss according to difficulty. However, there already exist several work
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning