Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations

TL;DR

This paper enhances understanding of Maximum Manifold Capacity Representations (MMCR) in multi-view self-supervised learning, revealing its geometric and information-theoretic properties, and demonstrates its effectiveness across different data modalities.

Contribution

It provides a theoretical and empirical analysis of MMCR, connecting geometric and information-theoretic perspectives, and introduces scaling laws for better utilization in MVSSL.

Findings

MMCR encourages alignment and uniformity in embeddings

Embeddings maximize mutual information lower bounds

MMCR performs well on multimodal image-text data

Abstract

Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is intriguing because it does not fit neatly into any of the commonplace MVSSL lineages, instead originating from a statistical mechanical perspective on the linear separability of data manifolds. In this paper, we seek to improve our understanding and our utilization of MMCR. To better understand MMCR, we leverage tools from high dimensional probability to demonstrate that MMCR incentivizes alignment and uniformity of learned embeddings. We then leverage tools from information theory to show that such embeddings maximize a well-known lower bound on mutual information between views, thereby connecting the geometric perspective of MMCR to the information-theoretic perspective commonly discussed in MVSSL. To better utilize MMCR, we mathematically predict and experimentally confirm non-monotonic changes in the pretraining loss akin to double descent but with respect to atypical hyperparameters. We also discover compute scaling laws that enable predicting the pretraining loss as a function of gradients steps, batch size, embedding dimension and number of views. We then show that MMCR, originally applied to image data, is performant on multimodal image-text data. By more deeply understanding the theoretical and empirical behavior of MMCR, our work reveals insights on improving MVSSL methods.