Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports

TL;DR

This paper presents a RAG-based system that automates the generation of situation awareness reports from diverse real-time data sources, enhancing speed and accuracy in humanitarian decision-making.

Contribution

It introduces a novel dynamic RAG framework with a three-level evaluation process for automated, reliable situation reports in peacebuilding contexts.

Findings

System effectively generates coherent, relevant reports in real-world scenarios.

Automates and accelerates situation awareness report production.

Evaluation framework improves report quality and reliability.

Abstract

Timely and accurate situation awareness is vital for decision-making in humanitarian response, conflict monitoring, and early warning and early action. However, the manual analysis of vast and heterogeneous data sources often results in delays, limiting the effectiveness of interventions. This paper introduces a dynamic Retrieval-Augmented Generation (RAG) system that autonomously generates situation awareness reports by integrating real-time data from diverse sources, including news articles, conflict event databases, and economic indicators. Our system constructs query-specific knowledge bases on demand, ensuring timely, relevant, and accurate insights. To ensure the quality of generated reports, we propose a three-level evaluation framework that combines semantic similarity metrics, factual consistency checks, and expert feedback. The first level employs automated NLP metrics to assess coherence and factual accuracy. The second level involves human expert evaluation to verify the relevance and completeness of the reports. The third level utilizes LLM-as-a-Judge, where large language models provide an additional layer of assessment to ensure robustness. The system is tested across multiple real-world scenarios, demonstrating its effectiveness in producing coherent, insightful, and actionable reports. By automating report generation, our approach reduces the burden on human analysts and accelerates decision-making processes. To promote reproducibility and further research, we openly share our code and evaluation tools with the community via GitHub.