Rule-Bottleneck Reinforcement Learning: Joint Explanation and Decision Optimization for Resource Allocation with Language Agents

TL;DR

This paper introduces Rule-Bottleneck Reinforcement Learning (RBRL), a framework combining decision-making and explanation generation using language models to improve transparency and efficiency in resource allocation tasks.

Contribution

RBRL jointly optimizes decision policies and explanations by integrating LLM-generated rules with RL, enhancing transparency and performance in resource allocation.

Findings

RBRL achieves competitive performance with deep RL methods.

RBRL offers efficiency gains over LLM fine-tuning.

Survey confirms improved explanation quality.

Abstract

Deep Reinforcement Learning (RL) is remarkably effective in addressing sequential resource allocation problems in domains such as healthcare, public policy, and resource management. However, deep RL policies often lack transparency and adaptability, challenging their deployment alongside human decision-makers. In contrast, Language Agents, powered by large language models (LLMs), provide human-understandable reasoning but may struggle with effective decision making. To bridge this gap, we propose Rule-Bottleneck Reinforcement Learning (RBRL), a novel framework that jointly optimizes decision and explanations. At each step, RBRL generates candidate rules with an LLM, selects among them using an attention-based RL policy, and determines the environment action with an explanation via chain-of-thought reasoning. The RL rule selection is optimized using the environment rewards and an explainability metric judged by the LLM. Evaluations in real-world scenarios highlight RBRL's competitive performance with deep RL and efficiency gains over LLM fine-tuning. A survey further confirms the enhanced quality of its explanations.