Orthogonality-Promoting Distance Metric Learning: Convex Relaxation and Theoretical Analysis

TL;DR

This paper introduces convex relaxations for orthogonality-promoting distance metric learning, providing theoretical insights and demonstrating improved effectiveness and efficiency over nonconvex approaches.

Contribution

It develops convex relaxations for nonconvex DML problems, offers theoretical analysis of OPR's role in balancedness and generalization, and empirically shows superior performance.

Findings

Convex relaxations guarantee global optimal solutions.

OPR promotes balancedness and compactness effectively.

Convex methods outperform nonconvex ones in experiments.

Abstract

Distance metric learning (DML), which learns a distance metric from labeled "similar" and "dissimilar" data pairs, is widely utilized. Recently, several works investigate orthogonality-promoting regularization (OPR), which encourages the projection vectors in DML to be close to being orthogonal, to achieve three effects: (1) high balancedness -- achieving comparable performance on both frequent and infrequent classes; (2) high compactness -- using a small number of projection vectors to achieve a "good" metric; (3) good generalizability -- alleviating overfitting to training data. While showing promising results, these approaches suffer three problems. First, they involve solving non-convex optimization problems where achieving the global optimal is NP-hard. Second, it lacks a theoretical understanding why OPR can lead to balancedness. Third, the current generalization error analysis of OPR is not directly on the regularizer. In this paper, we address these three issues by (1) seeking convex relaxations of the original nonconvex problems so that the global optimal is guaranteed to be achievable; (2) providing a formal analysis on OPR's capability of promoting balancedness; (3) providing a theoretical analysis that directly reveals the relationship between OPR and generalization performance. Experiments on various datasets demonstrate that our convex methods are more effective in promoting balancedness, compactness, and generalization, and are computationally more efficient, compared with the nonconvex methods.