Attraction-Repulsion Spectrum in Neighbor Embeddings

TL;DR

This paper explores how adjusting the balance between attraction and repulsion in neighbor embedding algorithms like t-SNE, UMAP, and ForceAtlas2 affects their ability to represent data structures, revealing a spectrum of embeddings with different properties.

Contribution

It introduces the attraction-repulsion spectrum in neighbor embeddings and explains how different algorithms correspond to different points on this spectrum, highlighting inherent trade-offs.

Findings

Stronger attraction improves continuous manifold representation.

Stronger repulsion enhances cluster separation and kNN recall.

UMAP and ForceAtlas2 correspond to increased attraction compared to t-SNE.

Abstract

Neighbor embeddings are a family of methods for visualizing complex high-dimensional datasets using $k$NN graphs. To find the low-dimensional embedding, these algorithms combine an attractive force between neighboring pairs of points with a repulsive force between all points. One of the most popular examples of such algorithms is t-SNE. Here we empirically show that changing the balance between the attractive and the repulsive forces in t-SNE using the exaggeration parameter yields a spectrum of embeddings, which is characterized by a simple trade-off: stronger attraction can better represent continuous manifold structures, while stronger repulsion can better represent discrete cluster structures and yields higher $k$NN recall. We find that UMAP embeddings correspond to t-SNE with increased attraction; mathematical analysis shows that this is because the negative sampling optimisation strategy employed by UMAP strongly lowers the effective repulsion. Likewise, ForceAtlas2, commonly used for visualizing developmental single-cell transcriptomic data, yields embeddings corresponding to t-SNE with the attraction increased even more. At the extreme of this spectrum lie Laplacian Eigenmaps. Our results demonstrate that many prominent neighbor embedding algorithms can be placed onto the attraction-repulsion spectrum, and highlight the inherent trade-offs between them.