PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use

TL;DR

PEARL is a novel framework that significantly improves large language models' ability to plan and execute complex multi-turn tool use through a two-stage learning process, enhancing robustness and success rates.

Contribution

Introducing PEARL, a two-stage reinforcement learning framework that enhances LLM planning and tool use, addressing prior challenges like hallucination and weak planning.

Findings

Achieves 56.5% success rate on ToolHop benchmark.

Reduces tool invocation error rate.

Outperforms existing methods significantly.

Abstract

Large Language Models show great potential with external tools, but face significant challenges in complex, multi-turn tool invocation. They often exhibit weak planning, tool hallucination, erroneous parameter generation, and struggle with robust interaction. To tackle these issues, we present PEARL, a novel framework to enhance LLM planning and execution for sophisticated tool use. PEARL adopts a two-stage approach: an offline phase where the agent explores tools to learn valid usage patterns and failure conditions, and an online reinforcement learning phase. In the online phase, a dedicated Planner is trained via group Relative Policy Optimization (GRPO) with a carefully designed reward function that provides distinct signals for planning quality. Experiments on the ToolHop and T-Eval benchmarks show PEARL significantly outperforms existing methods, achieving a new state-of-the-art success rate of \textbf{56.5\%} on ToolHop while maintaining a low invocation error rate. Our work marks a key advance in addressing the complex planning challenges of tool use, contributing to the development of more robust and reliable LLM-based agents.