Automated Document Indexing via Intelligent Hierarchical Clustering: A Novel Approach

TL;DR

This paper presents an automated, scalable hierarchical clustering framework for document organization that uses minimal input and statistical methods, demonstrating improved accuracy and robustness over traditional clustering techniques.

Contribution

It introduces a novel hierarchical clustering algorithm that automates hierarchy attribute determination and incorporates topic modeling for robust document indexing.

Findings

Outperforms K-Means and K-Medoids in F-measure

Scalable and effective for large document corpora

Automated hierarchy attribute determination enhances robustness

Abstract

With the rising quantity of textual data available in electronic format, the need to organize it become a highly challenging task. In the present paper, we explore a document organization framework that exploits an intelligent hierarchical clustering algorithm to generate an index over a set of documents. The framework has been designed to be scalable and accurate even with large corpora. The advantage of the proposed algorithm lies in the need for minimal inputs, with much of the hierarchy attributes being decided in an automated manner using statistical methods. The use of topic modeling in a pre-processing stage ensures robustness to a range of variations in the input data. For experimental work 20-Newsgroups dataset has been used. The F- measure of the proposed approach has been compared with the traditional K-Means and K-Medoids clustering algorithms. Test results demonstrate the applicability, efficiency and effectiveness of our proposed approach. After extensive experimentation, we conclude that the framework shows promise for further research and specialized commercial applications.