Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions

TL;DR

This paper introduces a customized knowledge distillation method called BCKD for efficient dense image prediction models on AI chips, focusing on boundary accuracy and region connectivity to improve performance.

Contribution

The paper proposes a novel boundary and context knowledge distillation approach tailored for EDIP models, enhancing boundary quality and region connectivity in compact models.

Findings

Improves boundary region completeness in student models.

Ensures target region connectivity through self-relation transfer.

Demonstrates effectiveness across multiple tasks and datasets.

Abstract

It has been revealed that efficient dense image prediction (EDIP) models designed for AI chips, trained using the knowledge distillation (KD) framework, encounter two key challenges, including \emph{maintaining boundary region completeness} and \emph{ensuring target region connectivity}, despite their favorable real-time capacity to recognize the main object regions. In this work, we propose a customized boundary and context knowledge distillation (BCKD) method for EDIPs, which facilitates the targeted KD from large accurate teacher models to compact small student models. Specifically, the \emph{boundary distillation} focuses on extracting explicit object-level boundaries from the hierarchical feature maps to enhance the student model's mask quality in boundary regions. Meanwhile, the \emph{context distillation} leverages self-relations as a bridge to transfer implicit pixel-level contexts from the teacher model to the student model, ensuring strong connectivity in target regions. Our proposed method is specifically designed for the EDIP tasks and is characterized by its simplicity and efficiency. Theoretical analysis and extensive experimental results across semantic segmentation, object detection, and instance segmentation on five representative datasets demonstrate the effectiveness of BCKD, resulting in well-defined object boundaries and smooth connecting regions.