An Analytical Approach to Document Clustering Based on Internal Criterion Function

TL;DR

This paper introduces a new document clustering algorithm based on an internal criterion function that aims to overcome common issues of local optima and empty clusters, demonstrating improved performance across multiple datasets.

Contribution

A novel document clustering algorithm utilizing an internal criterion function that converges to a global optimum and improves with more clusters.

Findings

Algorithm does not suffer from local optima or empty clusters.

Performance improves as the number of clusters increases.

Validated on three datasets with various cluster counts.

Abstract

Fast and high quality document clustering is an important task in organizing information, search engine results obtaining from user query, enhancing web crawling and information retrieval. With the large amount of data available and with a goal of creating good quality clusters, a variety of algorithms have been developed having quality-complexity trade-offs. Among these, some algorithms seek to minimize the computational complexity using certain criterion functions which are defined for the whole set of clustering solution. In this paper, we are proposing a novel document clustering algorithm based on an internal criterion function. Most commonly used partitioning clustering algorithms (e.g. k-means) have some drawbacks as they suffer from local optimum solutions and creation of empty clusters as a clustering solution. The proposed algorithm usually does not suffer from these problems and converge to a global optimum, its performance enhances with the increase in number of clusters. We have checked our algorithm against three different datasets for four different values of k (required number of clusters).