LLM-Powered Automatic Translation and Urgency in Crisis Scenarios

TL;DR

This paper evaluates the performance of large language models and translation systems in crisis scenarios, revealing significant issues in maintaining urgency and stability across languages, which poses risks for crisis communication effectiveness.

Contribution

It introduces a new urgency-annotated multilingual dataset and provides a comprehensive evaluation of LLMs and translation systems in crisis contexts, highlighting their limitations.

Findings

Both translation models and LLMs show performance degradation in crisis translation.

Urgency perception can be distorted even with accurate translations.

Urgency classification varies significantly with language and input.

Abstract

Large language models (LLMs) are increasingly proposed for crisis preparedness and response, particularly for multilingual communication. However, their suitability for high-stakes crisis contexts remains insufficiently evaluated. This work examines the performance of state-of-the-art LLMs and machine translation systems in crisis-domain translation, with a focus on preserving urgency, which is a critical property for effective crisis communication and triaging. Using multilingual crisis data and a newly introduced urgency-annotated dataset covering over 32 languages, we show that both dedicated translation models and LLMs exhibit substantial performance degradation and instability. Crucially, even linguistically adequate translations can distort perceived urgency, and LLM-based urgency classifications vary widely depending on the language of the prompt and input. These findings highlight significant risks in deploying general-purpose language technologies for crisis communication and underscore the need for crisis-aware evaluation frameworks.