EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning

TL;DR

This paper introduces Explicit Policy Optimization (EPO), a reinforcement learning-based method that enhances strategic reasoning in large language models, enabling better performance in complex, real-world scenarios like negotiations and navigation.

Contribution

The paper proposes EPO, a novel reinforcement learning framework that improves adaptability and transferability of strategic reasoning in LLMs through multi-turn RL and self-play.

Findings

Achieves state-of-the-art results in social dialogue and web navigation tasks.

Demonstrates emergent collaborative reasoning mechanisms in EPO.

Effectively generates novel strategies for complex reasoning tasks.

Abstract

Large Language Models (LLMs) have shown impressive reasoning capabilities in well-defined problems with clear solutions, such as mathematics and coding. However, they still struggle with complex real-world scenarios like business negotiations, which require strategic reasoning-an ability to navigate dynamic environments and align long-term goals amidst uncertainty. Existing methods for strategic reasoning face challenges in adaptability, scalability, and transferring strategies to new contexts. To address these issues, we propose explicit policy optimization (EPO) for strategic reasoning, featuring an LLM that provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. To improve adaptability and policy transferability, we train the strategic reasoning model via multi-turn reinforcement learning (RL),utilizing process rewards and iterative self-play. Experiments across social and physical domains demonstrate EPO's ability of long-term goal alignment through enhanced strategic reasoning, achieving state-of-the-art performance on social dialogue and web navigation tasks. Our findings reveal various collaborative reasoning mechanisms emergent in EPO and its effectiveness in generating novel strategies, underscoring its potential for strategic reasoning in real-world applications. Code and data are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/EPO.