Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing

TL;DR

This paper introduces a framework for creating transparent, auditable, and human-in-the-loop LM-powered subroutines, demonstrated through a public comment processing application for environmental reviews.

Contribution

It proposes a static typing framework for LM subroutines with auditability and online improvement, and implements it in a real-world public comment processing system.

Findings

The framework enables transparent auditing of LM artifacts.

The CommentNEPA system effectively summarizes public comments.

Quantitative evaluation shows alignment with human-annotated data.

Abstract

The advent of language models (LMs) has the potential to dramatically accelerate tasks that may be cast to text-processing; however, real-world adoption is hindered by concerns regarding safety, explainability, and bias. How can we responsibly leverage LMs in a transparent, auditable manner -- minimizing risk and allowing human experts to focus on informed decision-making rather than data-processing or prompt engineering? In this work, we propose a framework for declaring statically typed, LM-powered subroutines (i.e., callable, function-like procedures) for use within conventional asynchronous code -- such that sparse feedback from human experts is used to improve the performance of each subroutine online (i.e., during use). In our implementation, all LM-produced artifacts (i.e., prompts, inputs, outputs, and data-dependencies) are recorded and exposed to audit on demand. We package this framework as a library to support its adoption and continued development. While this framework may be applicable across several real-world decision workflows (e.g., in healthcare and legal fields), we evaluate it in the context of public comment processing as mandated by the 1969 National Environmental Protection Act (NEPA): Specifically, we use this framework to develop "CommentNEPA," an application that compiles, organizes, and summarizes a corpus of public commentary submitted in response to a project requiring environmental review. We quantitatively evaluate the application by comparing its outputs (when operating without human feedback) to historical ``ground-truth'' data as labelled by human annotators during the preparation of official environmental impact statements.