Poster: Self-Supervised Quantization-Aware Knowledge Distillation

TL;DR

This paper introduces SQAKD, a self-supervised framework for quantization-aware training that improves accuracy without relying on labeled data, by unifying quantization dynamics and jointly optimizing discretization and KL-loss.

Contribution

It proposes a novel self-supervised approach to QAT that unifies quantization dynamics and reframes the process as a co-optimization problem, enhancing performance and accessibility.

Findings

Significantly improves state-of-the-art QAT methods

Does not require extensive labeled data

Establishes stronger baseline performances

Abstract

Quantization-aware training (QAT) starts with a pre-trained full-precision model and performs quantization during retraining. However, existing QAT works require supervision from the labels and they suffer from accuracy loss due to reduced precision. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation framework (SQAKD). SQAKD first unifies the forward and backward dynamics of various quantization functions and then reframes QAT as a co-optimization problem that simultaneously minimizes the KL-Loss and the discretization error, in a self-supervised manner. The evaluation shows that SQAKD significantly improves the performance of various state-of-the-art QAT works. SQAKD establishes stronger baselines and does not require extensive labeled training data, potentially making state-of-the-art QAT research more accessible.