Non-isotropy Regularization for Proxy-based Deep Metric Learning

TL;DR

This paper introduces non-isotropy regularization ($ ext{NIR}$) for proxy-based Deep Metric Learning, improving local structure learning and generalization by enforcing unique sample-proxy translatability using Normalizing Flows, achieving state-of-the-art results.

Contribution

We propose $ ext{NIR}$, a novel regularization method leveraging Normalizing Flows to induce non-isotropic sample distributions around proxies, enhancing local structure learning in DML.

Findings

$ ext{NIR}$ improves generalization across benchmarks.

Achieves state-of-the-art performance on CUB200-2011, Cars196, and Stanford Online Products.

Retains or improves convergence properties of proxy-based methods.

Abstract

Deep Metric Learning (DML) aims to learn representation spaces on which semantic relations can simply be expressed through predefined distance metrics. Best performing approaches commonly leverage class proxies as sample stand-ins for better convergence and generalization. However, these proxy-methods solely optimize for sample-proxy distances. Given the inherent non-bijectiveness of used distance functions, this can induce locally isotropic sample distributions, leading to crucial semantic context being missed due to difficulties resolving local structures and intraclass relations between samples. To alleviate this problem, we propose non-isotropy regularization ($\mathbb{NIR}$) for proxy-based Deep Metric Learning. By leveraging Normalizing Flows, we enforce unique translatability of samples from their respective class proxies. This allows us to explicitly induce a non-isotropic distribution of samples around a proxy to optimize for. In doing so, we equip proxy-based objectives to better learn local structures. Extensive experiments highlight consistent generalization benefits of $\mathbb{NIR}$ while achieving competitive and state-of-the-art performance on the standard benchmarks CUB200-2011, Cars196 and Stanford Online Products. In addition, we find the superior convergence properties of proxy-based methods to still be retained or even improved, making $\mathbb{NIR}$ very attractive for practical usage. Code available at https://github.com/ExplainableML/NonIsotropicProxyDML.