Horseshoes in multidimensional scaling and local kernel methods

TL;DR

This paper analyzes how multidimensional scaling and local kernel methods produce horseshoe patterns in low-dimensional embeddings, especially when data has an underlying order and only local distances are accurate.

Contribution

It provides a rigorous analysis of horseshoe patterns in MDS and kernel methods, linking them to latent orderings and local information in manifold learning.

Findings

Horseshoe patterns arise from latent data orderings.

Local distance information suffices to produce horseshoes.

Results offer insights into manifold learning on curves.

Abstract

Classical multidimensional scaling (MDS) is a method for visualizing high-dimensional point clouds by mapping to low-dimensional Euclidean space. This mapping is defined in terms of eigenfunctions of a matrix of interpoint dissimilarities. In this paper we analyze in detail multidimensional scaling applied to a specific dataset: the 2005 United States House of Representatives roll call votes. Certain MDS and kernel projections output ``horseshoes'' that are characteristic of dimensionality reduction techniques. We show that, in general, a latent ordering of the data gives rise to these patterns when one only has local information. That is, when only the interpoint distances for nearby points are known accurately. Our results provide a rigorous set of results and insight into manifold learning in the special case where the manifold is a curve.