Contrastive Knowledge Amalgamation for Unsupervised Image Classification

TL;DR

This paper introduces Contrastive Knowledge Amalgamation (CKA), a novel framework that improves unsupervised image classification by aligning teacher-student models using contrastive and alignment losses, enhancing decision boundaries and class separation.

Contribution

The paper proposes a new CKA framework that employs contrastive and alignment losses to better align heterogeneous teacher-student models for unsupervised image classification.

Findings

CKA improves classification accuracy on benchmarks.

Contrastive and alignment losses enhance decision boundary learning.

Framework generalizes to multiple tasks and heterogeneous models.

Abstract

Knowledge amalgamation (KA) aims to learn a compact student model to handle the joint objective from multiple teacher models that are are specialized for their own tasks respectively. Current methods focus on coarsely aligning teachers and students in the common representation space, making it difficult for the student to learn the proper decision boundaries from a set of heterogeneous teachers. Besides, the KL divergence in previous works only minimizes the probability distribution difference between teachers and the student, ignoring the intrinsic characteristics of teachers. Therefore, we propose a novel Contrastive Knowledge Amalgamation (CKA) framework, which introduces contrastive losses and an alignment loss to achieve intra-class cohesion and inter-class separation.Contrastive losses intra- and inter- models are designed to widen the distance between representations of different classes. The alignment loss is introduced to minimize the sample-level distribution differences of teacher-student models in the common representation space.Furthermore, the student learns heterogeneous unsupervised classification tasks through soft targets efficiently and flexibly in the task-level amalgamation. Extensive experiments on benchmarks demonstrate the generalization capability of CKA in the amalgamation of specific task as well as multiple tasks. Comprehensive ablation studies provide a further insight into our CKA.