Approximating Metric Magnitude of Point Sets

TL;DR

This paper introduces efficient algorithms for approximating the metric magnitude of point sets, enabling its practical use in machine learning tasks like regularization and clustering.

Contribution

It presents two novel algorithms for fast approximation of metric magnitude, extending its application in machine learning and analyzing its relation to model generalization.

Findings

Approximation algorithms are fast and accurate.

Longer model sequences during training correlate with better generalization.

New applications include regularization and clustering in neural networks.

Abstract

Metric magnitude is a measure of the "size" of point clouds with many desirable geometric properties. It has been adapted to various mathematical contexts and recent work suggests that it can enhance machine learning and optimization algorithms. But its usability is limited due to the computational cost when the dataset is large or when the computation must be carried out repeatedly (e.g. in model training). In this paper, we study the magnitude computation problem, and show efficient ways of approximating it. We show that it can be cast as a convex optimization problem, but not as a submodular optimization. The paper describes two new algorithms - an iterative approximation algorithm that converges fast and is accurate, and a subset selection method that makes the computation even faster. It has been previously proposed that magnitude of model sequences generated during stochastic gradient descent is correlated to generalization gap. Extension of this result using our more scalable algorithms shows that longer sequences in fact bear higher correlations. We also describe new applications of magnitude in machine learning - as an effective regularizer for neural network training, and as a novel clustering criterion.