Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents

TL;DR

This paper introduces entropy reduction as a supervisory signal to optimize tool-use behavior in large language model agents, significantly reducing unnecessary tool calls and improving task performance.

Contribution

It proposes novel reward strategies based on entropy reduction, demonstrating substantial improvements in tool-use efficiency and effectiveness across various domains.

Findings

Entropy reduction correlates with high-quality tool calls.

Sparse rewards reduce tool calls by 72.07%.

Dense rewards improve task performance by 22.27%.

Abstract

Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.