Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification

TL;DR

This paper investigates the use of knowledge distillation from various deep neural network teachers to train a small CNN student for multi-script identification in natural scene images, achieving satisfactory results on CVSI-2015.

Contribution

It explores the effectiveness of dark knowledge transfer using LSTM and CNN-based teachers to improve a simple CNN student in multi-script identification.

Findings

Knowledge transfer improves student performance

LSTM and CNN teachers both enhance accuracy

Satisfactory results on CVSI-2015 dataset

Abstract

Multi-lingual script identification is a difficult task consisting of different language with complex backgrounds in scene text images. According to the current research scenario, deep neural networks are employed as teacher models to train a smaller student network by utilizing the teacher model's predictions. This process is known as dark knowledge transfer. It has been quite successful in many domains where the final result obtained is unachievable through directly training the student network with a simple architecture. In this paper, we explore dark knowledge transfer approach using long short-term memory(LSTM) and CNN based assistant model and various deep neural networks as the teacher model, with a simple CNN based student network, in this domain of multi-script identification from natural scene text images. We explore the performance of different teacher models and their ability to transfer knowledge to a student network. Although the small student network's limited size, our approach obtains satisfactory results on a well-known script identification dataset CVSI-2015.