The Information Theory of Similarity

TL;DR

This paper unifies similarity search methods with information theory, showing that similarity measures relate to mutual information and that fundamental limits govern encoding efficiency and ranking preservation.

Contribution

It establishes a rigorous mathematical equivalence between similarity systems and Shannon's information theory, deriving fundamental bounds and revealing the physical nature of semantic similarity.

Findings

REWA's complexity bound is proven optimal.

Similarity search relates to mutual information and channel capacity.

Fundamental limits constrain encoding and ranking preservation.

Abstract

We establish a precise mathematical equivalence between witness-based similarity systems (REWA) and Shannon's information theory. We prove that witness overlap is mutual information, that REWA bit complexity bounds arise from channel capacity limitations, and that ranking-preserving encodings obey rate-distortion constraints. This unification reveals that fifty years of similarity search research -- from Bloom filters to locality-sensitive hashing to neural retrieval -- implicitly developed information theory for relational data. We derive fundamental lower bounds showing that REWA's $O(\Delta^{-2} \log N)$ complexity is optimal: no encoding scheme can preserve similarity rankings with fewer bits. The framework establishes that semantic similarity has physical units (bits of mutual information), search is communication (query transmission over a noisy channel), and retrieval systems face fundamental capacity limits analogous to Shannon's channel coding theorem.