A Big Data Analytics System for Predicting Suicidal Ideation in Real-Time Based on Social Media Streaming Data

TL;DR

This paper presents a big data architecture combining batch and real-time processing to predict suicidal ideation from social media data, achieving high accuracy and enabling early intervention.

Contribution

It introduces a novel big data framework integrating batch and streaming analysis for real-time mental health prediction from social media content.

Findings

Achieved 93.47% accuracy with Unigram + Bigram + CV-IDF features and MLP classifier.

Demonstrated effective real-time prediction of suicidal ideation using Twitter streaming data.

Validated the approach's potential for early mental health intervention.

Abstract

Online social media platforms have recently become integral to our society and daily routines. Every day, users worldwide spend a couple of hours on such platforms, expressing their sentiments and emotional state and contacting each other. Analyzing such huge amounts of data from these platforms can provide a clear insight into public sentiments and help detect their mental status. The early identification of these health condition risks may assist in preventing or reducing the number of suicide ideation and potentially saving people's lives. The traditional techniques have become ineffective in processing such streams and large-scale datasets. Therefore, the paper proposed a new methodology based on a big data architecture to predict suicidal ideation from social media content. The proposed approach provides a practical analysis of social media data in two phases: batch processing and real-time streaming prediction. The batch dataset was collected from the Reddit forum and used for model building and training, while streaming big data was extracted using Twitter streaming API and used for real-time prediction. After the raw data was preprocessed, the extracted features were fed to multiple Apache Spark ML classifiers: NB, LR, LinearSVC, DT, RF, and MLP. We conducted various experiments using various feature-extraction techniques with different testing scenarios. The experimental results of the batch processing phase showed that the features extracted of (Unigram + Bigram) + CV-IDF with MLP classifier provided high performance for classifying suicidal ideation, with an accuracy of 93.47%, and then applied for real-time streaming prediction phase.