Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations

TL;DR

This paper presents a novel framework that uses flow matching with generative normalizing flows to train language models for explanations, improving alignment with human judgments and decision accuracy.

Contribution

It introduces a general-purpose reinforcement learning approach using distributional rewards from CNFs, capturing the probabilistic nature of human explanations and bounding deviations from true rewards.

Findings

Explanations generated improve prediction accuracy of agent decisions.

Explanations exhibit greater logical soundness and are more actionable.

Lower cognitive load compared to baseline methods.

Abstract

As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain agent policies in natural language is vital for reliable coexistence. We introduce a general-purpose framework that trains explanation-generating LLMs via reinforcement learning from AI feedback, with distributional rewards generated by generative continuous normalizing flows (CNFs). CNFs capture the pluralistic and probabilistic nature of human judgments about explanations. Moreover, under mild assumptions, CNFs provably bound deviations from true human reward distributions when trained on noisy proxy rewards from LLMs. We design a specialized CNF architecture that selectively attends to linguistic cues in the decision context and explanations when generating rewards. Human and LLM evaluators find that our method delivers explanations that enable more accurate predictions of true agent decisions, exhibit greater logical soundness and actionability, and impose lower cognitive load than explanations trained with proxy LLM rewards or state-of-the-art RLHF and RLAIF baselines.