A Role-Based LLM Framework for Structured Information Extraction from Healthy Food Policies

TL;DR

This paper introduces a role-based LLM framework that improves structured information extraction from healthy food policies by mimicking expert workflows and incorporating domain knowledge.

Contribution

The study presents a novel role-based LLM framework that enhances accuracy and transparency in extracting information from complex health policy documents.

Findings

Outperforms zero-shot, few-shot, and CoT baselines on policy data

Demonstrates superior performance in complex reasoning tasks

Provides a reliable, transparent methodology for policy information extraction

Abstract

Current Large Language Model (LLM) approaches for information extraction (IE) in the healthy food policy domain are often hindered by various factors, including misinformation, specifically hallucinations, misclassifications, and omissions that result from the structural diversity and inconsistency of policy documents. To address these limitations, this study proposes a role-based LLM framework that automates the IE from unstructured policy data by assigning specialized roles: an LLM policy analyst for metadata and mechanism classification, an LLM legal strategy specialist for identifying complex legal approaches, and an LLM food system expert for categorizing food system stages. This framework mimics expert analysis workflows by incorporating structured domain knowledge, including explicit definitions of legal mechanisms and classification criteria, into role-specific prompts. We evaluate the framework using 608 healthy food policies from the Healthy Food Policy Project (HFPP) database, comparing its performance against zero-shot, few-shot, and chain-of-thought (CoT) baselines using Llama-3.3-70B. Our proposed framework demonstrates superior performance in complex reasoning tasks, offering a reliable and transparent methodology for automating IE from health policies.