Scalable link prediction in Twitter using self-configured framework

TL;DR

This paper introduces a self-configuring framework for scalable link prediction in large social networks like Twitter, using machine learning to optimize parameters automatically, improving efficiency and resource utilization.

Contribution

The novel Self-Configured Framework (SCF) automates parameter tuning in Spark-based link prediction, enhancing scalability and reducing manual setup errors.

Findings

40% reduction in prediction time

Balanced resource consumption

Effective on large Twitter datasets

Abstract

Link prediction analysis becomes vital to acquire a deeper understanding of events underlying social networks interactions and connections especially in current evolving and large-scale social networks. Traditional link prediction approaches underperformed for most large-scale social networks in terms of its scalability and efficiency. Spark is a distributed open-source framework that facilitate scalable link prediction efficiency in large-scale social networks. The framework provides numerous tunable properties for users to manually configure the parameters for the applications. However, manual configurations open to performance issue when the applications start scaling tremendously, which is hard to set up and expose to human errors. This paper introduced a novel Self-Configured Framework (SCF) to provide an autonomous feature in Spark that predicts and sets the best configuration instantly before the application execution using XGBoost classifier. SCF is evaluated on the Twitter social network using three link prediction applications: Graph Clustering (GC), Overlapping Community Detection (OCD), and Redundant Graph Clustering (RGD) to assess the impact of shifting data sizes on different applications in Twitter. The result demonstrates a 40% reduction in prediction time as well as a balanced resource consumption that makes full use of resources, especially for limited number and size of clusters