Enhancing keyword correlation for event detection in social networks using SVD and k-means: Twitter case study

TL;DR

This paper introduces a method combining SVD and k-means clustering to improve the correlation of textual features from tweets with real-world events, enhancing event detection accuracy.

Contribution

The study presents a novel approach that applies SVD and clustering to strengthen feature signals and improve correlation with event data in social media analysis.

Findings

Correlation scores increased from 0.3 to 0.6 after applying the method.

Average correlation scores improved from 0.3 to 0.4.

Method is effective across multiple correlation techniques and feature types.

Abstract

Extracting textual features from tweets is a challenging process due to the noisy nature of the content and the weak signal of most of the words used. In this paper, we propose using singular value decomposition (SVD) with clustering to enhance the signals of the textual features in the tweets to improve the correlation with events. The proposed technique applies SVD to the time series vector for each feature to factorize the matrix of feature/day counts, in order to ensure the independence of the feature vectors. Afterwards, the k-means clustering is applied to build a look-up table that maps members of each cluster to the cluster-centroid. The lookup table is used to map each feature in the original data to the centroid of its cluster, then we calculate the sum of the term frequency vectors of all features in each cluster to the term-frequency-vector of the cluster centroid. To test the technique we calculated the correlations of the cluster centroids with the golden standard record (GSR) vector before and after summing the vectors of the cluster members to the centroid-vector. The proposed method is applied to multiple correlation techniques including the Pearson, Spearman, distance correlation and Kendal Tao. The experiments have also considered the different word forms and lengths of the features including keywords, n-grams, skip-grams and bags-of-words. The correlation results are enhanced significantly as the highest correlation scores have increased from 0.3 to 0.6, and the average correlation scores have increased from 0.3 to 0.4.