LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration

TL;DR

LumiNet introduces a perception-driven approach to improve logit-based knowledge distillation by calibrating logits based on model representation, effectively addressing overconfidence and outperforming feature-based methods on major benchmarks.

Contribution

The paper presents LumiNet, a novel logit-based distillation method that incorporates perception for better calibration and knowledge transfer, surpassing existing feature-based approaches.

Findings

Outperforms feature-based methods on CIFAR-100, ImageNet, and MSCOCO.

Achieves 1.5% and 2.05% improvements over KD with ResNet18 and MobileNetV2 on ImageNet.

Effectively addresses overconfidence in logit-based distillation.

Abstract

In the knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill "dark knowledge" from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of "perception", aiming to calibrate logits based on the model's representation capability. This concept addresses overconfidence issues in the logit-based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature-based methods, e.g., compared to KD with ResNet18 and MobileNetV2 on ImageNet, it shows improvements of 1.5% and 2.05%, respectively. Codes are available at https://github.com/ismail31416/LumiNet.