Compression-based Similarity

TL;DR

This paper introduces a novel similarity measure based on compression for binary objects and internet search counts for object names, grounded in Kolmogorov complexity, with extensive experimental validation.

Contribution

It presents a unified framework for measuring similarity using compression and search data, extending Kolmogorov complexity concepts to practical applications.

Findings

Effective similarity measures for binary files and object names.

Successful experimental validation of the proposed methods.

Demonstrated applicability to diverse data types.

Abstract

First we consider pair-wise distances for literal objects consisting of finite binary files. These files are taken to contain all of their meaning, like genomes or books. The distances are based on compression of the objects concerned, normalized, and can be viewed as similarity distances. Second, we consider pair-wise distances between names of objects, like "red" or "christianity." In this case the distances are based on searches of the Internet. Such a search can be performed by any search engine that returns aggregate page counts. We can extract a code length from the numbers returned, use the same formula as before, and derive a similarity or relative semantics between names for objects. The theory is based on Kolmogorov complexity. We test both similarities extensively experimentally.