Synergistic Self-supervised and Quantization Learning

TL;DR

This paper introduces SSQL, a self-supervised learning method that enhances the quantization friendliness of models, improving accuracy at low bit-widths and benefiting multiple downstream tasks without extra storage overhead.

Contribution

The paper proposes SSQL, a novel self-supervised pretraining approach that contrasts quantized and full precision features to improve low-bit quantization performance and model versatility.

Findings

Significantly improves accuracy of quantized models at low bit-widths

Boosts full precision model accuracy in most cases

Enables multiple downstream tasks with a single training process

Abstract

With the success of self-supervised learning (SSL), it has become a mainstream paradigm to fine-tune from self-supervised pretrained models to boost the performance on downstream tasks. However, we find that current SSL models suffer severe accuracy drops when performing low-bit quantization, prohibiting their deployment in resource-constrained applications. In this paper, we propose a method called synergistic self-supervised and quantization learning (SSQL) to pretrain quantization-friendly self-supervised models facilitating downstream deployment. SSQL contrasts the features of the quantized and full precision models in a self-supervised fashion, where the bit-width for the quantized model is randomly selected in each step. SSQL not only significantly improves the accuracy when quantized to lower bit-widths, but also boosts the accuracy of full precision models in most cases. By only training once, SSQL can then benefit various downstream tasks at different bit-widths simultaneously. Moreover, the bit-width flexibility is achieved without additional storage overhead, requiring only one copy of weights during training and inference. We theoretically analyze the optimization process of SSQL, and conduct exhaustive experiments on various benchmarks to further demonstrate the effectiveness of our method. Our code is available at https://github.com/megvii-research/SSQL-ECCV2022.