A Generalization Theory of Cross-Modality Distillation with Contrastive Learning

TL;DR

This paper introduces a theoretical framework for cross-modality contrastive distillation, providing convergence analysis and demonstrating improved performance across various modalities and tasks.

Contribution

It formulates a general contrastive distillation framework and offers the first convergence analysis linking modality distance to downstream task error.

Findings

Outperforms existing methods by 2-3% across multiple modalities.

Provides theoretical insights into the impact of modality distance on test error.

Validates the framework through extensive experiments on recognition and segmentation tasks.

Abstract

Cross-modality distillation arises as an important topic for data modalities containing limited knowledge such as depth maps and high-quality sketches. Such techniques are of great importance, especially for memory and privacy-restricted scenarios where labeled training data is generally unavailable. To solve the problem, existing label-free methods leverage a few pairwise unlabeled data to distill the knowledge by aligning features or statistics between the source and target modalities. For instance, one typically aims to minimize the L2 distance or contrastive loss between the learned features of pairs of samples in the source (e.g. image) and the target (e.g. sketch) modalities. However, most algorithms in this domain only focus on the experimental results but lack theoretical insight. To bridge the gap between the theory and practical method of cross-modality distillation, we first formulate a general framework of cross-modality contrastive distillation (CMCD), built upon contrastive learning that leverages both positive and negative correspondence, towards a better distillation of generalizable features. Furthermore, we establish a thorough convergence analysis that reveals that the distance between source and target modalities significantly impacts the test error on downstream tasks within the target modality which is also validated by the empirical results. Extensive experimental results show that our algorithm outperforms existing algorithms consistently by a margin of 2-3\% across diverse modalities and tasks, covering modalities of image, sketch, depth map, and audio and tasks of recognition and segmentation.