Analysis and study on text representation to improve the accuracy of the Normalized Compression Distance

TL;DR

This paper investigates how various distortion techniques affect the Normalized Compression Distance (NCD) to enhance text similarity measurement accuracy, aiming to better understand text and compression distance properties.

Contribution

It provides an analysis of the impact of distortion techniques on NCD, offering insights to improve text comparison methods using compression-based distances.

Findings

Distortion techniques significantly influence NCD accuracy

Certain distortions improve the robustness of NCD for text comparison

Insights into the nature of texts and compression distances are gained

Abstract

The huge amount of information stored in text form makes methods that deal with texts really interesting. This thesis focuses on dealing with texts using compression distances. More specifically, the thesis takes a small step towards understanding both the nature of texts and the nature of compression distances. Broadly speaking, the way in which this is done is exploring the effects that several distortion techniques have on one of the most successful distances in the family of compression distances, the Normalized Compression Distance -NCD-.