The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes

TL;DR

This paper demonstrates that dense low-dimensional representations in information retrieval degrade in performance more rapidly than sparse methods as index size grows, especially at very low dimensions, challenging their assumed superiority.

Contribution

It provides a theoretical and empirical analysis revealing the limitations of dense low-dimensional representations at large index sizes, highlighting a potential performance tipping point.

Findings

Dense representations' performance declines faster than sparse ones with increasing index size.

Lower-dimensional dense representations are more prone to false positives.

Sparse representations can outperform dense ones beyond a certain index size.

Abstract

Information Retrieval using dense low-dimensional representations recently became popular and showed out-performance to traditional sparse-representations like BM25. However, no previous work investigated how dense representations perform with large index sizes. We show theoretically and empirically that the performance for dense representations decreases quicker than sparse representations for increasing index sizes. In extreme cases, this can even lead to a tipping point where at a certain index size sparse representations outperform dense representations. We show that this behavior is tightly connected to the number of dimensions of the representations: The lower the dimension, the higher the chance for false positives, i.e. returning irrelevant documents.