Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making

TL;DR

This paper introduces ACE, a co-evolution framework combining LLMs and RL to improve large-scale decision-making, addressing their individual limitations through mutual refinement and high-quality data sharing.

Contribution

The paper proposes a novel co-evolution framework where LLMs and RL agents mutually enhance each other for large-scale decision tasks, a significant advancement over prior isolated approaches.

Findings

ACE outperforms existing RL and LLM-based methods in power grid management tasks.

The dual-role trajectory refinement improves decision quality and efficiency.

Large action spaces exceeding 60K are effectively handled by ACE.

Abstract

Recent advancements in Large Language Models (LLMs) and Reinforcement Learning (RL) have shown significant promise in decision-making tasks. Nevertheless, for large-scale industrial decision problems, both approaches face distinct challenges: LLMs lack real-time long-sequence decision-making capabilities, while RL struggles with sample efficiency in vast action spaces. To bridge this gap, we propose Agents Co-Evolution (ACE), a synergistic framework between LLMs and RL agents for large-scale decision-making scenarios. ACE introduces a dual-role trajectory refinement mechanism where LLMs act as both Policy Actor and Value Critic during RL's training: the Actor refines suboptimal actions via multi-step reasoning and environment validation, while the Critic performs temporal credit assignment through trajectory-level reward shaping. Concurrently, RL agent enhances LLMs' task-specific decision-making with high-quality fine-tuning datasets generated via prioritized experience replay. Through extensive experiments across multiple power grid operation challenges with action spaces exceeding 60K discrete actions, ACE demonstrates superior performance over existing RL methods and LLM-based methods.