Generalization Bounds for Metric and Similarity Learning

TL;DR

This paper develops new theoretical generalization bounds for metric and similarity learning methods, linking the bounds to matrix norms and Rademacher complexities, and highlighting advantages of sparse regularization.

Contribution

It introduces a novel analysis framework reducing generalization bounds to Rademacher averages over sample blocks, with specific bounds for different matrix norms including sparse regularization.

Findings

Sparse $L^1$-norm regularization yields better generalization bounds.

Generalization bounds are derived using Rademacher complexity and U-statistics techniques.

Analysis applies to various matrix-norm regularizers in metric and similarity learning.

Abstract

Recently, metric learning and similarity learning have attracted a large amount of interest. Many models and optimisation algorithms have been proposed. However, there is relatively little work on the generalization analysis of such methods. In this paper, we derive novel generalization bounds of metric and similarity learning. In particular, we first show that the generalization analysis reduces to the estimation of the Rademacher average over "sums-of-i.i.d." sample-blocks related to the specific matrix norm. Then, we derive generalization bounds for metric/similarity learning with different matrix-norm regularisers by estimating their specific Rademacher complexities. Our analysis indicates that sparse metric/similarity learning with $L^1$-norm regularisation could lead to significantly better bounds than those with Frobenius-norm regularisation. Our novel generalization analysis develops and refines the techniques of U-statistics and Rademacher complexity analysis.