Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning

TL;DR

This paper introduces a novel self-distillation framework with online diffusion on batch manifolds that enhances deep metric learning by capturing intrinsic data relationships, leading to improved performance across multiple benchmarks.

Contribution

It proposes a new online batch diffusion process integrated with progressive self-distillation, effectively capturing local geometric structures to improve deep metric learning.

Findings

Consistently improves state-of-the-art DML methods.

Achieves significant performance gains on CUB200, CARS196, and Stanford Online Products.

Requires negligible additional training time.

Abstract

Recent deep metric learning (DML) methods typically leverage solely class labels to keep positive samples far away from negative ones. However, this type of method normally ignores the crucial knowledge hidden in the data (e.g., intra-class information variation), which is harmful to the generalization of the trained model. To alleviate this problem, in this paper we propose Online Batch Diffusion-based Self-Distillation (OBD-SD) for DML. Specifically, we first propose a simple but effective Progressive Self-Distillation (PSD), which distills the knowledge progressively from the model itself during training. The soft distance targets achieved by PSD can present richer relational information among samples, which is beneficial for the diversity of embedding representations. Then, we extend PSD with an Online Batch Diffusion Process (OBDP), which is to capture the local geometric structure of manifolds in each batch, so that it can reveal the intrinsic relationships among samples in the batch and produce better soft distance targets. Note that our OBDP is able to restore the insufficient manifold relationships obtained by the original PSD and achieve significant performance improvement. Our OBD-SD is a flexible framework that can be integrated into state-of-the-art (SOTA) DML methods. Extensive experiments on various benchmarks, namely CUB200, CARS196, and Stanford Online Products, demonstrate that our OBD-SD consistently improves the performance of the existing DML methods on multiple datasets with negligible additional training time, achieving very competitive results. Code: \url{https://github.com/ZelongZeng/OBD-SD_Pytorch}