Insights into Ordinal Embedding Algorithms: A Systematic Evaluation

TL;DR

This paper systematically evaluates various ordinal embedding algorithms, revealing that simple non-convex methods outperform more complex approaches, including neural networks, due to their resilience to local optima.

Contribution

It provides the first comprehensive empirical comparison of ordinal embedding algorithms and introduces a unified GPU-accelerated library for large-scale embedding tasks.

Findings

Non-convex methods outperform complex algorithms.

Simple methods are more scalable and robust.

Neural network approaches do not outperform traditional methods.

Abstract

The objective of ordinal embedding is to find a Euclidean representation of a set of abstract items, using only answers to triplet comparisons of the form "Is item $i$ closer to the item $j$ or item $k$?". In recent years, numerous algorithms have been proposed to solve this problem. However, there does not exist a fair and thorough assessment of these embedding methods and therefore several key questions remain unanswered: Which algorithms perform better when the embedding dimension is constrained or few triplet comparisons are available? Which ones scale better with increasing sample size or dimension? In our paper, we address these questions and provide the first comprehensive and systematic empirical evaluation of existing algorithms as well as a new neural network approach. We find that simple, relatively unknown, non-convex methods consistently outperform all other algorithms, including elaborate approaches based on neural networks or landmark approaches. This finding can be explained by our insight that many of the non-convex optimization approaches do not suffer from local optima. Our comprehensive assessment is enabled by our unified library of popular embedding algorithms that leverages GPU resources and allows for fast and accurate embeddings of millions of data points.