Multimodal Learning with Augmentation Techniques for Natural Disaster Assessment

TL;DR

This paper investigates augmentation techniques for improving multimodal disaster assessment models using social media data, addressing class imbalance and limited samples with diffusion, back-translation, and paraphrasing methods.

Contribution

It introduces novel augmentation strategies for visual and textual data in multimodal disaster assessment, demonstrating improved classification performance on the CrisisMMD dataset.

Findings

Augmentation improves performance for underrepresented classes

Diffusion-based methods enhance visual data quality

Multi-view learning shows potential but needs refinement

Abstract

Natural disaster assessment relies on accurate and rapid access to information, with social media emerging as a valuable real-time source. However, existing datasets suffer from class imbalance and limited samples, making effective model development a challenging task. This paper explores augmentation techniques to address these issues on the CrisisMMD multimodal dataset. For visual data, we apply diffusion-based methods, namely Real Guidance and DiffuseMix. For text data, we explore back-translation, paraphrasing with transformers, and image caption-based augmentation. We evaluated these across unimodal, multimodal, and multi-view learning setups. Results show that selected augmentations improve classification performance, particularly for underrepresented classes, while multi-view learning introduces potential but requires further refinement. This study highlights effective augmentation strategies for building more robust disaster assessment systems.