Establishing a stronger baseline for lightweight contrastive models

TL;DR

This paper improves lightweight contrastive learning models by optimizing training settings and introducing a smoothed loss, achieving performance close to larger models without needing a pretrained teacher.

Contribution

It establishes a stronger baseline for lightweight contrastive models by tailoring training recipes and proposing a smoothed InfoNCE loss, eliminating the need for pretrained teacher models.

Findings

Significant accuracy improvements on ImageNet for MobileNet-V3-Large and EfficientNet-B0.

Achieved close-to-resNet50 performance with 5x fewer parameters.

Proposed a smoothed InfoNCE loss to reduce noise in contrastive learning.

Abstract

Recent research has reported a performance degradation in self-supervised contrastive learning for specially designed efficient networks, such as MobileNet and EfficientNet. A common practice to address this problem is to introduce a pretrained contrastive teacher model and train the lightweight networks with distillation signals generated by the teacher. However, it is time and resource consuming to pretrain a teacher model when it is not available. In this work, we aim to establish a stronger baseline for lightweight contrastive models without using a pretrained teacher model. Specifically, we show that the optimal recipe for efficient models is different from that of larger models, and using the same training settings as ResNet50, as previous research does, is inappropriate. Additionally, we observe a common issu e in contrastive learning where either the positive or negative views can be noisy, and propose a smoothed version of InfoNCE loss to alleviate this problem. As a result, we successfully improve the linear evaluation results from 36.3\% to 62.3\% for MobileNet-V3-Large and from 42.2\% to 65.8\% for EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with $5\times$ fewer parameters. We hope our research will facilitate the usage of lightweight contrastive models.