Real Time Sentiment Change Detection of Twitter Data Streams

TL;DR

This paper presents a real-time, efficient method for detecting sentiment changes in Twitter data streams using open source tools, control charts, and lexicon-based sentiment analysis, suitable for large-scale monitoring.

Contribution

It introduces a novel real-time sentiment change detection approach that is memory-efficient and computationally inexpensive, suitable for big data Twitter streams.

Findings

Successfully detects sentiment shifts in real-time Twitter data

Efficiently handles high volume and velocity of streaming data

Potential application in early fake news and propaganda detection

Abstract

In the past few years, there has been a huge growth in Twitter sentiment analysis having already provided a fair amount of research on sentiment detection of public opinion among Twitter users. Given the fact that Twitter messages are generated constantly with dizzying rates, a huge volume of streaming data is created, thus there is an imperative need for accurate methods for knowledge discovery and mining of this information. Although there exists a plethora of twitter sentiment analysis methods in the recent literature, the researchers have shifted to real-time sentiment identification on twitter streaming data, as expected. A major challenge is to deal with the Big Data challenges arising in Twitter streaming applications concerning both Volume and Velocity. Under this perspective, in this paper, a methodological approach based on open source tools is provided for real-time detection of changes in sentiment that is ultra efficient with respect to both memory consumption and computational cost. This is achieved by iteratively collecting tweets in real time and discarding them immediately after their process. For this purpose, we employ the Lexicon approach for sentiment characterizations, while change detection is achieved through appropriate control charts that do not require historical information. We believe that the proposed methodology provides the trigger for a potential large-scale monitoring of threads in an attempt to discover fake news spread or propaganda efforts in their early stages. Our experimental real-time analysis based on a recent hashtag provides evidence that the proposed approach can detect meaningful sentiment changes across a hashtags lifetime.