Cross-Modal Knowledge Transfer Without Task-Relevant Source Data

TL;DR

This paper introduces SOCKET, a novel method for transferring knowledge from a source model trained on RGB data to a target modality like depth or infrared, without access to source data, by reducing modality gap using paired data and feature normalization.

Contribution

The paper presents SOCKET, the first approach for source-free cross-modal knowledge transfer that effectively handles modality gaps without relying on source data.

Findings

SOCKET outperforms existing methods in classification tasks involving different modalities.

The method effectively reduces the modality gap using paired data and feature normalization.

Extensive experiments demonstrate significant improvements over prior source-free approaches.

Abstract

Cost-effective depth and infrared sensors as alternatives to usual RGB sensors are now a reality, and have some advantages over RGB in domains like autonomous navigation and remote sensing. As such, building computer vision and deep learning systems for depth and infrared data are crucial. However, large labeled datasets for these modalities are still lacking. In such cases, transferring knowledge from a neural network trained on a well-labeled large dataset in the source modality (RGB) to a neural network that works on a target modality (depth, infrared, etc.) is of great value. For reasons like memory and privacy, it may not be possible to access the source data, and knowledge transfer needs to work with only the source models. We describe an effective solution, SOCKET: SOurce-free Cross-modal KnowledgE Transfer for this challenging task of transferring knowledge from one source modality to a different target modality without access to task-relevant source data. The framework reduces the modality gap using paired task-irrelevant data, as well as by matching the mean and variance of the target features with the batch-norm statistics that are present in the source models. We show through extensive experiments that our method significantly outperforms existing source-free methods for classification tasks which do not account for the modality gap.