Differential Attention for Multimodal Crisis Event Analysis

TL;DR

This paper introduces a novel multimodal fusion approach using Differential Attention and Guided Cross Attention with vision-language models to improve crisis event classification from noisy social media data.

Contribution

It proposes combining CLIP embeddings, LLaVA-generated text, and adaptive fusion strategies to enhance multimodal crisis data analysis without task-specific fine-tuning.

Findings

Outperforms traditional models on CrisisMMD dataset

Differential Attention improves classification accuracy

Guided Cross Attention effectively aligns multimodal features

Abstract

Social networks can be a valuable source of information during crisis events. In particular, users can post a stream of multimodal data that can be critical for real-time humanitarian response. However, effectively extracting meaningful information from this large and noisy data stream and effectively integrating heterogeneous data remains a formidable challenge. In this work, we explore vision language models (VLMs) and advanced fusion strategies to enhance the classification of crisis data in three different tasks. We incorporate LLaVA-generated text to improve text-image alignment. Additionally, we leverage Contrastive Language-Image Pretraining (CLIP)-based vision and text embeddings, which, without task-specific fine-tuning, outperform traditional models. To further refine multimodal fusion, we employ Guided Cross Attention (Guided CA) and combine it with the Differential Attention mechanism to enhance feature alignment by emphasizing critical information while filtering out irrelevant content. Our results show that while Differential Attention improves classification performance, Guided CA remains highly effective in aligning multimodal features. Extensive experiments on the CrisisMMD benchmark data set demonstrate that the combination of pretrained VLMs, enriched textual descriptions, and adaptive fusion strategies consistently outperforms state-of-the-art models in classification accuracy, contributing to more reliable and interpretable models for three different tasks that are crucial for disaster response. Our code is available at https://github.com/Munia03/Multimodal_Crisis_Event.