Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

TL;DR

This paper introduces a graph-based knowledge distillation framework that uses multi-level feature alignment and spectral embedding to improve the transfer of complex structural information from teacher to student models in visual tasks.

Contribution

It proposes a novel graph knowledge distillation method with spectral embedding, capturing relational and structural knowledge for better student model performance.

Findings

Outperforms previous feature distillation methods on CIFAR-100, MS-COCO, and Pascal VOC datasets.

Effectively captures complex structural dependencies through graph-based representation.

Enhances student model performance by preserving relational knowledge from teacher models.

Abstract

In visual tasks, large teacher models capture essential features and deep information, enhancing performance. However, distilling this information into smaller student models often leads to performance loss due to structural differences and capacity limitations. To tackle this, we propose a distillation framework based on graph knowledge, including a multi-level feature alignment strategy and an attention-guided mechanism to provide a targeted learning trajectory for the student model. We emphasize spectral embedding (SE) as a key technique in our distillation process, which merges the student's feature space with the relational knowledge and structural complexities similar to the teacher network. This method captures the teacher's understanding in a graph-based representation, enabling the student model to more accurately mimic the complex structural dependencies present in the teacher model. Compared to methods that focus only on specific distillation areas, our strategy not only considers key features within the teacher model but also endeavors to capture the relationships and interactions among feature sets, encoding these complex pieces of information into a graph structure to understand and utilize the dynamic relationships among these pieces of information from a global perspective. Experiments show that our method outperforms previous feature distillation methods on the CIFAR-100, MS-COCO, and Pascal VOC datasets, proving its efficiency and applicability.