Efficient Clustering from Distributions over Topics

TL;DR

This paper introduces a novel clustering method that leverages topic modeling to efficiently identify similar documents in large corpora, outperforming existing techniques in speed while maintaining accuracy.

Contribution

The paper proposes a topic-based clustering approach that reduces computational costs by focusing similarity computations on smaller, relevant subsets of documents.

Findings

Outperforms state-of-the-art clustering techniques in efficiency by over 50%.

Effective in identifying similar scientific publications.

Reduces computational time significantly without sacrificing accuracy.

Abstract

There are many scenarios where we may want to find pairs of textually similar documents in a large corpus (e.g. a researcher doing literature review, or an R&D project manager analyzing project proposals). To programmatically discover those connections can help experts to achieve those goals, but brute-force pairwise comparisons are not computationally adequate when the size of the document corpus is too large. Some algorithms in the literature divide the search space into regions containing potentially similar documents, which are later processed separately from the rest in order to reduce the number of pairs compared. However, this kind of unsupervised methods still incur in high temporal costs. In this paper, we present an approach that relies on the results of a topic modeling algorithm over the documents in a collection, as a means to identify smaller subsets of documents where the similarity function can then be computed. This approach has proved to obtain promising results when identifying similar documents in the domain of scientific publications. We have compared our approach against state of the art clustering techniques and with different configurations for the topic modeling algorithm. Results suggest that our approach outperforms (> 0.5) the other analyzed techniques in terms of efficiency.