Mutual Regression Distance

TL;DR

This paper introduces the Mutual Regression Distance (MRD), a novel data manifold-aware metric that improves distribution comparison in machine learning tasks by being computationally efficient and theoretically robust.

Contribution

The paper proposes a new manifold-exploiting distance called MRD, with variants and algorithms, demonstrating its effectiveness and efficiency over existing measures like Wasserstein distance.

Findings

MRD is a pseudometric satisfying most metric axioms.

Simplified MRD has lower computational complexity than Wasserstein distance.

Numerical results show MRD's superiority in clustering, generative modeling, and domain adaptation.

Abstract

The maximum mean discrepancy and Wasserstein distance are popular distance measures between distributions and play important roles in many machine learning problems such as metric learning, generative modeling, domain adaption, and clustering. However, since they are functions of pair-wise distances between data points in two distributions, they do not exploit the potential manifold properties of data such as smoothness and hence are not effective in measuring the dissimilarity between the two distributions in the form of manifolds. In this paper, different from existing measures, we propose a novel distance called Mutual Regression Distance (MRD) induced by a constrained mutual regression problem, which can exploit the manifold property of data. We prove that MRD is a pseudometric that satisfies almost all the axioms of a metric. Since the optimization of the original MRD is costly, we provide a tight MRD and a simplified MRD, based on which a heuristic algorithm is established. We also provide kernel variants of MRDs that are more effective in handling nonlinear data. Our MRDs especially the simplified MRDs have much lower computational complexity than the Wasserstein distance. We provide theoretical guarantees, such as robustness, for MRDs. Finally, we apply MRDs to distribution clustering, generative models, and domain adaptation. The numerical results demonstrate the effectiveness and superiority of MRDs compared to the baselines.