Social Media Analysis for Crisis Informatics in the Cloud

TL;DR

This paper proposes a cloud-based infrastructure for analyzing social media data during crises, focusing on fast, reliable, and scalable data processing and Twitter stream collection, with comparisons to existing systems.

Contribution

Introduces a novel cloud infrastructure for crisis social media analysis that improves reliability, scalability, and ease of maintenance over existing solutions.

Findings

Enhanced reliability of Twitter data collection using container orchestration

Improved query performance with cloud-based infrastructure

Better scalability and extensibility compared to traditional systems

Abstract

Social media analysis of disaster events is a critical task in crisis informatics research. It involves analyzing social media data generated during natural disasters, crisis events, or other mass convergence events. Due to the large data sets generated during these events, large scale software infrastructures need to be designed to analyze the data in a timely manner. Creating such infrastructures bring the need to maintain them and this becomes more difficult as these infrastructures grow larger and older. Maintenance costs are high since there is a need for queries to be handled quickly which require large amounts of computational resources to be available on demand 24 hours a day, seven days a week. In this thesis, I describe an alternative approach to designing a software infrastructure for analyzing unstructured data on the cloud while providing fast queries and with the reliability needed for crisis informatics research. Additionally, I discuss a new approach for a more reliable Twitter stream collection using container orchestrated systems. I finally compare this new infrastructure with existing crisis informatics software infrastructures and compare their reliability, scalability and extensibility with my approach and my prototype.