Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge

TL;DR

This paper investigates how to optimize knowledge distillation for CNNs and Vision Transformers on edge devices by analyzing architecture choices, model size, image resolution, and fine-tuning effects.

Contribution

It provides a comprehensive analysis of factors affecting KD effectiveness on edge devices, including architecture, model size, image resolution, and post-distillation fine-tuning.

Findings

CNNs outperform ViTs in low-resource settings

Larger student models improve accuracy but reduce speed

Higher resolution images increase accuracy and memory usage

Abstract

This paper discusses four facets of the Knowledge Distillation (KD) process for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures, particularly when executed on edge devices with constrained processing capabilities. First, we conduct a comparative analysis of the KD process between CNNs and ViT architectures, aiming to elucidate the feasibility and efficacy of employing different architectural configurations for the teacher and student, while assessing their performance and efficiency. Second, we explore the impact of varying the size of the student model on accuracy and inference speed, while maintaining a constant KD duration. Third, we examine the effects of employing higher resolution images on the accuracy, memory footprint and computational workload. Last, we examine the performance improvements obtained by fine-tuning the student model after KD to specific downstream tasks. Through empirical evaluations and analyses, this research provides AI practitioners with insights into optimal strategies for maximizing the effectiveness of the KD process on edge devices.