A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method

TL;DR

This paper introduces CrisisSpot, a graph-based multimodal learning framework that incorporates social context features and a novel attention mechanism to improve disaster content classification during emergencies, validated on a new large dataset and existing benchmarks.

Contribution

It proposes CrisisSpot, a novel graph neural network model with IDEA attention and social context features, addressing limitations of prior methods in multimodal disaster content classification.

Findings

Achieved 9.45% F1-score improvement on CrisisMMD dataset.

Achieved 5.01% F1-score improvement on TSEqD dataset.

Demonstrated effectiveness of social context features and IDEA attention mechanism.

Abstract

In times of crisis, the prompt and precise classification of disaster-related information shared on social media platforms is crucial for effective disaster response and public safety. During such critical events, individuals use social media to communicate, sharing multimodal textual and visual content. However, due to the significant influx of unfiltered and diverse data, humanitarian organizations face challenges in leveraging this information efficiently. Existing methods for classifying disaster-related content often fail to model users' credibility, emotional context, and social interaction information, which are essential for accurate classification. To address this gap, we propose CrisisSpot, a method that utilizes a Graph-based Neural Network to capture complex relationships between textual and visual modalities, as well as Social Context Features to incorporate user-centric and content-centric information. We also introduce Inverted Dual Embedded Attention (IDEA), which captures both harmonious and contrasting patterns within the data to enhance multimodal interactions and provide richer insights. Additionally, we present TSEqD (Turkey-Syria Earthquake Dataset), a large annotated dataset for a single disaster event, containing 10,352 samples. Through extensive experiments, CrisisSpot demonstrated significant improvements, achieving an average F1-score gain of 9.45% and 5.01% compared to state-of-the-art methods on the publicly available CrisisMMD dataset and the TSEqD dataset, respectively.