Keep your distance: learning dispersed embeddings on $\mathbb{S}_m$

TL;DR

This paper explores methods for learning well-separated, dispersed embeddings on hyperspheres, connecting mathematical theory with practical algorithms to improve feature separation in high-dimensional spaces for tasks like image and text classification.

Contribution

It introduces new perspectives on pairwise dispersion, proposes an online Lloyd's algorithm variant, and develops a novel hypersphere-specific dispersion method, bridging theory and practice.

Findings

Dispersion improves classification performance in experiments.

Different algorithms show trade-offs depending on the regime.

Hypersphere-specific methods outperform general approaches in certain tasks.

Abstract

Learning well-separated features in high-dimensional spaces, such as text or image embeddings, is crucial for many machine learning applications. Achieving such separation can be effectively accomplished through the dispersion of embeddings, where unrelated vectors are pushed apart as much as possible. By constraining features to be on a hypersphere, we can connect dispersion to well-studied problems in mathematics and physics, where optimal solutions are known for limited low-dimensional cases. However, in representation learning we typically deal with a large number of features in high-dimensional space, and moreover, dispersion is usually traded off with some other task-oriented training objective, making existing theoretical and numerical solutions inapplicable. Therefore, it is common to rely on gradient-based methods to encourage dispersion, usually by minimizing some function of the pairwise distances. In this work, we first give an overview of existing methods from disconnected literature, making new connections and highlighting similarities. Next, we introduce some new angles. We propose to reinterpret pairwise dispersion using a maximum mean discrepancy (MMD) motivation. We then propose an online variant of the celebrated Lloyd's algorithm, of K-Means fame, as an effective alternative regularizer for dispersion on generic domains. Finally, we derive a novel dispersion method that directly exploits properties of the hypersphere. Our experiments show the importance of dispersion in image classification and natural language processing tasks, and how algorithms exhibit different trade-offs in different regimes.