Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design

TL;DR

This paper compares two methods of integrating expert labels into LLMs for detecting emission reduction goals in corporate reports, finding automatic prompt optimization to be more effective than example selection.

Contribution

It introduces a comparison between dynamic example selection and automatic prompt optimization for LLMs in emission goal detection tasks.

Findings

Automatic prompt optimization outperforms example selection.

Combining both methods offers limited additional benefit.

Optimized prompts capture task intricacies effectively.

Abstract

We address the detection of emission reduction goals in corporate reports, an important task for monitoring companies' progress in addressing climate change. Specifically, we focus on the issue of integrating expert feedback in the form of labeled example passages into LLM-based pipelines, and compare the two strategies of (1) a dynamic selection of few-shot examples and (2) the automatic optimization of the prompt by the LLM itself. Our findings on a public dataset of 769 climate-related passages from real-world business reports indicate that automatic prompt optimization is the superior approach, while combining both methods provides only limited benefit. Qualitative results indicate that optimized prompts do indeed capture many intricacies of the targeted emission goal extraction task.