Beyond Classification: Knowledge Distillation using Multi-Object Impressions

TL;DR

This paper introduces a novel zero-shot knowledge distillation method for object detection that synthesizes pseudo-data using only a pretrained teacher network, enabling effective student training without access to real training data.

Contribution

It presents the first approach for zero-data knowledge distillation in object detection by generating pseudo-samples from a pretrained teacher, bypassing the need for training data.

Findings

Achieves 64.2% mAP on KITTI with no training samples.

Outperforms baseline methods in zero-shot object detection distillation.

Demonstrates effectiveness across multiple benchmark datasets.

Abstract

Knowledge Distillation (KD) utilizes training data as a transfer set to transfer knowledge from a complex network (Teacher) to a smaller network (Student). Several works have recently identified many scenarios where the training data may not be available due to data privacy or sensitivity concerns and have proposed solutions under this restrictive constraint for the classification task. Unlike existing works, we, for the first time, solve a much more challenging problem, i.e., "KD for object detection with zero knowledge about the training data and its statistics". Our proposed approach prepares pseudo-targets and synthesizes corresponding samples (termed as "Multi-Object Impressions"), using only the pretrained Faster RCNN Teacher network. We use this pseudo-dataset as a transfer set to conduct zero-shot KD for object detection. We demonstrate the efficacy of our proposed method through several ablations and extensive experiments on benchmark datasets like KITTI, Pascal and COCO. Our approach with no training samples, achieves a respectable mAP of 64.2% and 55.5% on the student with same and half capacity while performing distillation from a Resnet-18 Teacher of 73.3% mAP on KITTI.