Real-time Text Analytics Pipeline Using Open-source Big Data Tools

TL;DR

This paper presents a real-time text analytics pipeline utilizing open-source big data tools to efficiently process large-scale social media data with minimal latency, enabling timely sentiment analysis.

Contribution

The paper introduces a novel real-time text processing pipeline combining Kafka, Spark, Cassandra, and D3, optimized for low latency and evaluated on Twitter data.

Findings

Processed 466,700 Tweets in 10.7 minutes with less than a minute latency.

Pipeline effectively minimizes latency in high-throughput text data processing.

Demonstrated scalability across different deployment scenarios.

Abstract

Real-time text processing systems are required in many domains to quickly identify patterns, trends, sentiments, and insights. Nowadays, social networks, e-commerce stores, blogs, scientific experiments, and server logs are main sources generating huge text data. However, to process huge text data in real time requires building a data processing pipeline. The main challenge in building such pipeline is to minimize latency to process high-throughput data. In this paper, we explain and evaluate our proposed real-time text processing pipeline using open-source big data tools which minimize the latency to process data streams. Our proposed data processing pipeline is based on Apache Kafka for data ingestion, Apache Spark for in-memory data processing, Apache Cassandra for storing processed results, and D3 JavaScript library for visualization. We evaluate the effectiveness of the proposed pipeline under varying deployment scenarios to perform sentiment analysis using Twitter dataset. Our experimental evaluations show less than a minute latency to process $466,700$ Tweets in $10.7$ minutes when three virtual machines allocated to the proposed pipeline.