Dimension Free Generalization Bounds for Non Linear Metric Learning

TL;DR

This paper establishes dimension-free generalization bounds for neural network-based metric learning, applicable in both sparse and non-sparse regimes, explaining generalization in practical, high-dimensional settings.

Contribution

It introduces novel uniform generalization bounds for metric learning with neural networks, covering non-sparse solutions via a new property called bounded amplification.

Findings

Bounds apply to both sparse and non-sparse regimes

Dimension-free guarantees explain generalization in high-dimensional data

Experimental validation on MNIST and 20newsgroups datasets

Abstract

In this work we study generalization guarantees for the metric learning problem, where the metric is induced by a neural network type embedding of the data. Specifically, we provide uniform generalization bounds for two regimes -- the sparse regime, and a non-sparse regime which we term \emph{bounded amplification}. The sparse regime bounds correspond to situations where $\ell_1$-type norms of the parameters are small. Similarly to the situation in classification, solutions satisfying such bounds can be obtained by an appropriate regularization of the problem. On the other hand, unregularized SGD optimization of a metric learning loss typically does not produce sparse solutions. We show that despite this lack of sparsity, by relying on a different, new property of the solutions, it is still possible to provide dimension free generalization guarantees. Consequently, these bounds can explain generalization in non sparse real experimental situations. We illustrate the studied phenomena on the MNIST and 20newsgroups datasets.