A Comparison of Document Similarity Algorithms

TL;DR

This paper evaluates various document similarity algorithms across three categories—statistical, neural networks, and knowledge-based—to identify the most effective methods for NLP applications like plagiarism detection and summarization.

Contribution

It categorizes and compares the most effective document similarity algorithms in each category using benchmark datasets and comprehensive evaluations.

Findings

Neural network algorithms outperform others in accuracy.

Statistical algorithms are faster but less accurate.

Knowledge-based algorithms excel in domain-specific tasks.

Abstract

Document similarity is an important part of Natural Language Processing and is most commonly used for plagiarism-detection and text summarization. Thus, finding the overall most effective document similarity algorithm could have a major positive impact on the field of Natural Language Processing. This report sets out to examine the numerous document similarity algorithms, and determine which ones are the most useful. It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms: statistical algorithms, neural networks, and corpus/knowledge-based algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations that test every possible area that each algorithm could be used in.