PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization

TL;DR

This paper introduces P-code Plans, a pseudocode-style reasoning framework that improves the generalization and efficiency of LLM agents, and proposes PGPO, a planning-guided preference optimization method that enhances agent performance.

Contribution

The paper presents P-code Plans for structured reasoning and PGPO, a novel training method that significantly boosts LLM agent reasoning quality and generalization.

Findings

PGPO outperforms current baselines on benchmark tasks.

P-code Plans improve reasoning efficiency and error reduction.

PGPO enhances high-quality plan generation and decision accuracy.

Abstract

Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents' ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style Planning Guided Preference Optimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents' ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning.