How to Choose Your Teacher for Fine Grained Image Recognition

TL;DR

This paper proposes a new teacher selection metric for knowledge distillation in fine-grained image recognition, significantly improving student model accuracy and efficiency.

Contribution

It introduces the Ratio 1-2 metric for better teacher selection, validated through extensive experiments across multiple datasets and models.

Findings

The metric improves teacher selection accuracy by 18%.

Small student models gain up to 17% in accuracy.

The codebase is publicly available at the provided GitHub URL.

Abstract

Fine-grained image recognition classifies subcategories such as bird species or car models. While state-of-the-art (SOTA) models are accurate, they are often too resource-intensive for deployment on constrained devices. Knowledge distillation addresses this by transferring knowledge from a large teacher model to a smaller student model. A key challenge is selecting the right teacher, as it heavily impacts student performance. This paper introduces a teacher selection metric, \textbf{Ratio 1-2}, based on teacher prediction ratios. Extensive analysis of over one thousand experiments across 3 students, 8 teachers, and 8 datasets under 4 training strategies demonstrates that our metric improves teacher selection by 18\% over previous methods, enabling small student models to achieve up to 17\% accuracy gains. Experiment codebase is available at: \href{https://github.com/arkel23/FGIR-KD-Teacher}{https://github.com/arkel23/FGIR-KD-Teacher}.