Responsible Retrieval Augmented Generation for Climate Decision Making from Documents

TL;DR

This paper develops a domain-specific evaluation framework and tools for responsible use of retrieval-augmented generation in climate decision-making, addressing challenges of accuracy, trust, and domain relevance.

Contribution

It introduces a novel evaluation framework, a human-annotated dataset, and automated tools tailored for climate-related RAG applications, promoting responsible deployment.

Findings

Key components for responsible RAG deployment identified

Insights into user experience and trust-building in high-risk domains

Evaluation framework and tools facilitate robust assessment in climate AI

Abstract

Climate decision making is constrained by the complexity and inaccessibility of key information within lengthy, technical, and multi-lingual documents. Generative AI technologies offer a promising route for improving the accessibility of information contained within these documents, but suffer from limitations. These include (1) a tendency to hallucinate or mis-represent information, (2) difficulty in steering or guaranteeing properties of generated output, and (3) reduced performance in specific technical domains. To address these challenges, we introduce a novel evaluation framework with domain-specific dimensions tailored for climate-related documents. We then apply this framework to evaluate Retrieval-Augmented Generation (RAG) approaches and assess retrieval- and generation-quality within a prototype tool that answers questions about individual climate law and policy documents. In addition, we publish a human-annotated dataset and scalable automated evaluation tools, with the aim of facilitating broader adoption and robust assessment of these systems in the climate domain. Our findings highlight the key components of responsible deployment of RAG to enhance decision-making, while also providing insights into user experience (UX) considerations for safely deploying such systems to build trust with users in high-risk domains.