A Lightweight LLM Framework for Disaster Humanitarian Information Classification

TL;DR

This paper presents a lightweight, cost-effective LLM framework for classifying disaster-related social media information, emphasizing parameter-efficient fine-tuning and evaluating various strategies to optimize performance in resource-constrained emergency scenarios.

Contribution

It introduces a unified disaster tweet dataset and systematically evaluates prompting, LoRA, QLoRA, and RAG strategies, demonstrating effective lightweight fine-tuning methods for crisis classification.

Findings

LoRA achieves 79.62% accuracy with only 2% of parameters.

QLoRA maintains performance with 50% memory usage.

RAG strategies degrade model performance due to label noise.

Abstract

Timely classification of humanitarian information from social media is critical for effective disaster response. However, deploying large language models (LLMs) for this task faces challenges in resource-constrained emergency settings. This paper develops a lightweight, cost-effective framework for disaster tweet classification using parameter-efficient fine-tuning. We construct a unified experimental corpus by integrating and normalizing the HumAID dataset (76,484 tweets across 19 disaster events) into a dual-task benchmark: humanitarian information categorization and event type identification. Through systematic evaluation of prompting strategies, LoRA fine-tuning, and retrieval-augmented generation (RAG) on Llama 3.1 8B, we demonstrate that: (1) LoRA achieves 79.62% humanitarian classification accuracy (+37.79% over zero-shot) while training only ~2% of parameters; (2) QLoRA enables efficient deployment with 99.4% of LoRA performance at 50% memory cost; (3) contrary to common assumptions, RAG strategies degrade fine-tuned model performance due to label noise from retrieved examples. These findings establish a practical, reproducible pipeline for building reliable crisis intelligence systems with limited computational resources.