Building Understandable Messaging for Policy and Evidence Review (BUMPER) with AI

TL;DR

This paper presents BUMPER, a framework leveraging large language models to create understandable, trustworthy messaging for policy and evidence review, aiming to improve evidence translation into policy with transparency and reliability.

Contribution

The paper introduces BUMPER, a novel framework that integrates LLMs with scientific knowledge bases to enhance trustworthiness and transparency in policy-relevant scientific communication.

Findings

Demonstrated a health policy example for measles control.

Showed how transparency and uncertainty measures improve trust.

Addressed reliability challenges of LLMs in high-stakes contexts.

Abstract

We introduce a framework for the use of large language models (LLMs) in Building Understandable Messaging for Policy and Evidence Review (BUMPER). LLMs are proving capable of providing interfaces for understanding and synthesizing large databases of diverse media. This presents an exciting opportunity to supercharge the translation of scientific evidence into policy and action, thereby improving livelihoods around the world. However, these models also pose challenges related to access, trust-worthiness, and accountability. The BUMPER framework is built atop a scientific knowledge base (e.g., documentation, code, survey data) by the same scientists (e.g., individual contributor, lab, consortium). We focus on a solution that builds trustworthiness through transparency, scope-limiting, explicit-checks, and uncertainty measures. LLMs are rapidly being adopted and consequences are poorly understood. The framework addresses open questions regarding the reliability of LLMs and their use in high-stakes applications. We provide a worked example in health policy for a model designed to inform measles control programs. We argue that this framework can facilitate accessibility of and confidence in scientific evidence for policymakers, drive a focus on policy-relevance and translatability for researchers, and ultimately increase and accelerate the impact of scientific knowledge used for policy decisions.