Towards Scalable Topic Detection on Web via Simulating Levy Walks Nature of Topics in Similarity Space

TL;DR

This paper introduces a novel Levy walk-inspired Explore-Exploit clustering method for scalable web topic detection, effectively distinguishing popular topics from noise webpages with improved efficiency and competitive accuracy.

Contribution

The paper proposes a new Levy walk-based Explore-Exploit approach for web topic clustering, enhancing scalability and noise filtering compared to existing methods.

Findings

Outperforms state-of-the-art in efficiency

Comparable effectiveness to existing methods

Effectively filters noise webpages

Abstract

Organizing a few webpages from social media websites into popular topics is one of the key steps to understand trends on web. Discovering popular topics from web faces a sea of noise webpages which never evolve into popular topics. In this paper, we discover that the similarity values between webpages in a popular topic contain the statistically similar features observed in Levy walks. Consequently, we present a simple, novel, yet very powerful Explore-Exploit (EE) approach to group topics by simulating Levy walks nature in the similarity space. The proposed EE-based topic clustering is an effective and effcient method which is a solid move towards handling a sea of noise webpages. Experiments on two public data sets demonstrate that our approach is not only comparable to the state-of-the-art methods in terms of effectiveness but also significantly outperforms the state-of-the-art methods in terms of efficiency.