Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning

TL;DR

This paper introduces a hierarchical framework called Agent-as-Tool that separates tool calling from reasoning in reinforcement learning agents, improving reasoning efficiency and performance in complex tasks.

Contribution

The study proposes a novel hierarchical approach that decouples tool calling from reasoning, enhancing reasoning clarity and efficiency in LLM-based reinforcement learning agents.

Findings

Achieved 63.2% exact match on Bamboogle, surpassing previous methods.

Only slight reinforcement fine-tuning needed on 180 samples.

Outperformed Search-R1 by 4.8% in exact match.

Abstract

Large Language Models (LLMs) have emerged as one of the most significant technological advancements in artificial intelligence in recent years. Their ability to understand, generate, and reason with natural language has transformed how we interact with AI systems. With the development of LLM-based agents and reinforcement-learning-based reasoning models, the study of applying reinforcement learning in agent frameworks has become a new research focus. However, all previous studies face the challenge of deciding the tool calling process and the reasoning process simultaneously, and the chain of reasoning was solely relied on the unprocessed raw result with redundant information and symbols unrelated to the task from the tool, which impose a heavy burden on the model's capability to reason. Therefore, in our research, we proposed a hierarchical framework Agent-as-tool that detach the tool calling process and the reasoning process, which enables the model to focus on the verbally reasoning process while the tool calling process is handled by another agent. Our work had achieved comparable results with only a slight reinforcement fine-tuning on 180 samples, and had achieved exceptionally well performance in Bamboogle with 63.2% of exact match and 75.2% in cover exact match, exceeding Search-R1 by 4.8% in exact match and 3.2% in cover exact match.