Report-based Recommendations for Policy Making and Agency Operations: Dataset and LLM Evaluation

TL;DR

This paper introduces a new benchmark for using large language models to generate policy recommendations from reports, aiming to support organizational decision-making and policy improvements.

Contribution

It presents the first dataset and evaluation framework for recommendation generation tailored to policy and agency operations, distinct from traditional recommendation systems.

Findings

State-of-the-art LLMs can effectively highlight key issues in reports.

The benchmark enables systematic evaluation of LLMs for policy recommendation tasks.

LLMs show potential to assist in policy analysis and organizational decision-making.

Abstract

Large Language Models (LLMs) are extensively used in text generation tasks. These generative capabilities bring us to a point where LLMs could potentially provide useful insights in policy making or agency operations. In this paper, we introduce a new task consisting of generating recommendations which can be used to inform future actions and improvements of agencies work within private and public organisations. In particular, we present the first benchmark and coherent evaluation for developing recommendation systems to inform organisation policies. This task is clearly different from usual product or user recommendation systems, but rather aims at providing a basis to suggest policy improvements based on the conclusions drawn from reports. Our results demonstrate that state-of-the-art LLMs have the potential to emphasize and reflect on key issues and learning points within generated recommendations.