Knowledge Distillation via Weighted Ensemble of Teaching Assistants

TL;DR

This paper proposes a novel knowledge distillation method that uses a weighted ensemble of multiple teaching assistants optimized by a differential evaluation algorithm to improve the performance of small student models.

Contribution

It introduces a weighted ensemble approach of multiple teaching assistants with optimized weights to enhance knowledge transfer in model compression.

Findings

Improved student model accuracy with multiple teaching assistants.

Effective weight optimization using differential evaluation algorithm.

Enhanced knowledge distillation performance over single assistant methods.

Abstract

Knowledge distillation in machine learning is the process of transferring knowledge from a large model called the teacher to a smaller model called the student. Knowledge distillation is one of the techniques to compress the large network (teacher) to a smaller network (student) that can be deployed in small devices such as mobile phones. When the network size gap between the teacher and student increases, the performance of the student network decreases. To solve this problem, an intermediate model is employed between the teacher model and the student model known as the teaching assistant model, which in turn bridges the gap between the teacher and the student. In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved. We combined these multiple teaching assistant models using weighted ensemble learning where we have used a differential evaluation optimization algorithm to generate the weight values.