Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners

TL;DR

This paper introduces Hybrid Distillation, a method that combines contrastive learning and masked image modeling teachers to enhance feature discrimination and diversity in representation learning.

Contribution

It proposes a novel Hybrid Distillation strategy that jointly leverages CL and MIM teachers, improving model performance by balancing discrimination and diversity.

Findings

Hybrid Distill outperforms previous methods on multiple benchmarks.

The approach effectively balances global pattern recognition and local attention.

Progressive token masking reduces training costs and improves robustness.

Abstract

Representation learning has been evolving from traditional supervised training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous works have demonstrated their pros and cons in specific scenarios, i.e., CL and supervised pre-training excel at capturing longer-range global patterns and enabling better feature discrimination, while MIM can introduce more local and diverse attention across all transformer layers. In this paper, we explore how to obtain a model that combines their strengths. We start by examining previous feature distillation and mask feature reconstruction methods and identify their limitations. We find that their increasing diversity mainly derives from the asymmetric designs, but these designs may in turn compromise the discrimination ability. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy, which utilizes both the supervised/CL teacher and the MIM teacher to jointly guide the student model. Hybrid Distill imitates the token relations of the MIM teacher to alleviate attention collapse, as well as distills the feature maps of the supervised/CL teacher to enable discrimination. Furthermore, a progressive redundant token masking strategy is also utilized to reduce the distilling costs and avoid falling into local optima. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.