Which is better? A Modularized Evaluation for Topic Popularity Prediction

TL;DR

This paper proposes a unified evaluation scheme for topic popularity prediction models in social networks, enabling better comparison and selection of methods based on comprehensive metrics and scenario-specific performance.

Contribution

It introduces a comprehensive evaluation framework with four modules and analyzes feature importance, aiding researchers in method comparison and improvement.

Findings

Feature-oriented methods excel in high-accuracy scenarios.

Relation-based methods offer more consistent performance.

The evaluation scheme improves comparability of different models.

Abstract

Topic popularity prediction in social networks has drawn much attention recently. Various elegant models have been proposed for this issue. However, different datasets and evaluation metrics they use lead to low comparability. So far there is no unified scheme to evaluate them, making it difficult to select and compare models. We conduct a comprehensible survey, propose an evaluation scheme and apply it to existing methods. Our scheme consists of four modules: classification; qualitative evaluation on several metrics; quantitative experiment on real world data; final ranking with risk matrix and $\textit{MinDis}$ to reflect performances under different scenarios. Furthermore, we analyze the efficiency and contribution of features used in feature oriented methods. The results show that feature oriented methods are more suitable for scenarios requiring high accuracy, while relation based methods have better consistency. Our work helps researchers compare and choose methods appropriately, and provides insights for further improvements.