A Weighted Similarity Metric for Community Detection in Sparse Data

TL;DR

This paper introduces a weighted similarity measure designed for community detection in highly sparse data, avoiding imputation and improving clustering quality in short text analysis.

Contribution

It proposes a novel weighted similarity metric that captures both presence and absence of features without data imputation, enhancing community detection in sparse datasets.

Findings

Outperforms traditional similarity measures in community detection tasks.

Provides consistent and robust results across different metrics and complexities.

Improves clustering quality in short consumer reviews.

Abstract

Many Natural Language Processing (NLP) related applications involves topics and sentiments derived from short documents such as consumer reviews and social media posts. Topics and sentiments of short documents are highly sparse because a short document generally covers a few topics among hundreds of candidates. Imputation of missing data is sometimes hard to justify and also often unpractical in highly sparse data. We developed a method for calculating a weighted similarity for highly sparse data without imputation. This weighted similarity is consist of three components to capture similarities based on both existence and lack of common properties and pattern of missing values. As a case study, we used a community detection algorithm and this weighted similarity to group different shampoo brands based on sparse topic sentiments derived from short consumer reviews. Compared with traditional imputation and similarity measures, the weighted similarity shows better performance in both general community structures and average community qualities. The performance is consistent and robust across metrics and community complexities.