Leveraging Topological Guidance for Improved Knowledge Distillation

TL;DR

This paper introduces TGD, a framework that uses topological features in knowledge distillation to improve lightweight image classification models, addressing computational challenges of TDA.

Contribution

The paper proposes a novel TGD framework that integrates topological features from multiple teachers into KD, enhancing performance and robustness in a computationally efficient manner.

Findings

TGD improves accuracy of lightweight models in image classification.

The approach enhances robustness against data perturbations.

Empirical results validate the effectiveness of topological guidance.

Abstract

Deep learning has shown its efficacy in extracting useful features to solve various computer vision tasks. However, when the structure of the data is complex and noisy, capturing effective information to improve performance is very difficult. To this end, topological data analysis (TDA) has been utilized to derive useful representations that can contribute to improving performance and robustness against perturbations. Despite its effectiveness, the requirements for large computational resources and significant time consumption in extracting topological features through TDA are critical problems when implementing it on small devices. To address this issue, we propose a framework called Topological Guidance-based Knowledge Distillation (TGD), which uses topological features in knowledge distillation (KD) for image classification tasks. We utilize KD to train a superior lightweight model and provide topological features with multiple teachers simultaneously. We introduce a mechanism for integrating features from different teachers and reducing the knowledge gap between teachers and the student, which aids in improving performance. We demonstrate the effectiveness of our approach through diverse empirical evaluations.