Generalization analysis with deep ReLU networks for metric and similarity learning

TL;DR

This paper provides the first explicit excess risk bounds for metric and similarity learning using structured deep ReLU networks, linking network complexity to generalization performance.

Contribution

It derives explicit excess risk bounds for metric and similarity learning by analyzing structured deep ReLU networks approximating the true metric.

Findings

Derived explicit excess risk rate based on network capacity.

Constructed structured deep ReLU networks approximating the true metric.

Empirical results show competitive performance and better similarity structure capture.

Abstract

While metric and similarity learning has been extensively studied from several theoretical perspectives, a rigorous understanding of its generalization performance is still lacking. In this paper, we investigate the generalization behavior of metric and similarity learning by exploiting the specific structure of the true metric (i.e., the target function). In particular, by deriving the explicit form of the true metric for metric and similarity learning with the hinge loss, we construct a structured deep ReLU neural network as an approximation of the true metric, whose approximation ability depends on the network complexity. Here, the network complexity is characterized by the network depth, the number of nonzero weights, and the number of computational units. Based on the hypothesis space consisting of such structured deep ReLU networks, we establish excess risk bounds for metric and similarity learning by carefully controlling both the approximation error and the estimation error. An explicit excess risk rate is derived by choosing the proper capacity of the constructed hypothesis space. To the best of our knowledge, this is the first generalization analysis that provides explicit excess risk bounds for metric and similarity learning. In addition, we investigate properties of the true metric for metric and similarity learning under more general loss functions. Experiments show that the proposed model is empirically competitive and better captures the underlying similarity structure.