# Knowledge Distillation in Object Detection: A Survey from CNN to Transformer

## TL;DR

This survey reviews knowledge distillation techniques for object detection, focusing on compressing large models for efficient deployment on resource-limited devices.

## Contribution

A comprehensive analysis of KD-based object detection methods from CNNs to Transformers, including their applications and limitations.

## Key findings

- Knowledge distillation effectively reduces model size and computational cost while maintaining accuracy.
- The survey highlights extended applications of KD in lightweight models and small object detection.
- KD is also applied in image classification, semantic segmentation, and 3D reconstruction.

## Abstract

Deep learning models, especially for object detection have gained immense popularity in computer vision. These models have demonstrated remarkable accuracy and performance, driving advancements across various applications. However, the high computational complexity and large storage requirements of state-of-the-art object detection models pose significant challenges for deployment on resource-constrained devices like mobile phones and embedded systems. Knowledge Distillation (KD) has emerged as a prominent solution to these challenges, effectively compressing large, complex teacher models into smaller, efficient student models. This technique maintains good accuracy while significantly reducing model size and computational demands, making object detection models more practical for real-world applications. This survey provides a comprehensive review of KD-based object detection models developed in recent years. It offers an in-depth analysis of existing techniques, highlighting their novelty and limitations, and explores future research directions. The survey covers the different distillation algorithms used in object detection. It also examines extended applications of knowledge distillation in object detection, such as improvements for lightweight models, addressing catastrophic forgetting in incremental learning, and enhancing small object detection. Furthermore, the survey also delves into the application of knowledge distillation in other domains such as image classification, semantic segmentation, 3D reconstruction, and document analysis.