Improving Generalization of Metric Learning via Listwise Self-distillation

TL;DR

This paper introduces Listwise Self-Distillation (LSD), a regularization technique for deep metric learning that improves generalization by adaptively refining distance targets and encouraging smoother embeddings, especially in challenging scenarios.

Contribution

The paper proposes LSD, a novel regularization method that enhances deep metric learning by leveraging self-distillation to better utilize internal sample relationships.

Findings

LSD consistently improves performance across multiple datasets.

LSD reduces overfitting in deep metric learning models.

LSD can be integrated into various existing DML frameworks.

Abstract

Most deep metric learning (DML) methods employ a strategy that forces all positive samples to be close in the embedding space while keeping them away from negative ones. However, such a strategy ignores the internal relationships of positive (negative) samples and often leads to overfitting, especially in the presence of hard samples and mislabeled samples. In this work, we propose a simple yet effective regularization, namely Listwise Self-Distillation (LSD), which progressively distills a model's own knowledge to adaptively assign a more appropriate distance target to each sample pair in a batch. LSD encourages smoother embeddings and information mining within positive (negative) samples as a way to mitigate overfitting and thus improve generalization. Our LSD can be directly integrated into general DML frameworks. Extensive experiments show that LSD consistently boosts the performance of various metric learning methods on multiple datasets.