Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval

TL;DR

This paper introduces a Capacity Dynamic Distillation framework that starts with a heavy student model and gradually compresses it during training, achieving faster inference and higher accuracy in image retrieval tasks.

Contribution

It proposes a novel dynamic capacity adjustment method for student models in knowledge distillation, enhancing efficiency without sacrificing performance.

Findings

Achieves 67.13% parameter reduction and 65.67% FLOPs reduction on VeRi-776 dataset.

Outperforms state-of-the-art methods in inference speed and accuracy.

Maintains around 2.11% higher mAP compared to existing approaches.

Abstract

Previous Knowledge Distillation based efficient image retrieval methods employs a lightweight network as the student model for fast inference. However, the lightweight student model lacks adequate representation capacity for effective knowledge imitation during the most critical early training period, causing final performance degeneration. To tackle this issue, we propose a Capacity Dynamic Distillation framework, which constructs a student model with editable representation capacity. Specifically, the employed student model is initially a heavy model to fruitfully learn distilled knowledge in the early training epochs, and the student model is gradually compressed during the training. To dynamically adjust the model capacity, our dynamic framework inserts a learnable convolutional layer within each residual block in the student model as the channel importance indicator. The indicator is optimized simultaneously by the image retrieval loss and the compression loss, and a retrieval-guided gradient resetting mechanism is proposed to release the gradient conflict. Extensive experiments show that our method has superior inference speed and accuracy, e.g., on the VeRi-776 dataset, given the ResNet101 as a teacher, our method saves 67.13% model parameters and 65.67% FLOPs (around 24.13% and 21.94% higher than state-of-the-arts) without sacrificing accuracy (around 2.11% mAP higher than state-of-the-arts).