Transferring Inter-Class Correlation

TL;DR

This paper introduces a novel knowledge transfer method in teacher-student frameworks for classification, utilizing self-attention based inter-class correlation maps to improve student network performance.

Contribution

It proposes a new inter-class correlation transfer method using self-attention maps, addressing the challenge of defining effective knowledge transfer.

Findings

Demonstrates improved classification accuracy with ICCT

Shows effectiveness across multiple neural network architectures

Provides a new perspective on knowledge transfer in T-S frameworks

Abstract

The Teacher-Student (T-S) framework is widely utilized in the classification tasks, through which the performance of one neural network (the student) can be improved by transferring knowledge from another trained neural network (the teacher). Since the transferring knowledge is related to the network capacities and structures between the teacher and the student, how to define efficient knowledge remains an open question. To address this issue, we design a novel transferring knowledge, the Self-Attention based Inter-Class Correlation (ICC) map in the output layer, and propose our T-S framework, Inter-Class Correlation Transfer (ICCT).