Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents

TL;DR

This paper introduces Calibrate-Then-Act, a framework enabling LLM agents to explicitly reason about cost-uncertainty tradeoffs, leading to more optimal decision-making in environment-interaction tasks.

Contribution

It formalizes cost-aware decision-making in LLM agents and proposes a method to improve their performance by inferring environment state priors before acting.

Findings

CTA improves decision strategies in synthetic, QA, and file reading tasks.

Explicit cost-benefit reasoning leads to more environment-sensitive agent behavior.

Agents using CTA outperform baseline approaches in tested scenarios.

Abstract

LLM agents are deployed in environments where they must interact to acquire information. In these scenarios, the agent must reason about inherent cost-uncertainty tradeoffs in how to act, such as when to stop exploring and commit to an answer. For instance, on a programming task, an agent might run the code it generates, or it might generate tests for that code snippet; the cost of writing and running a test is nonzero, but typically lower than the cost of running buggy code. In this work, we show that we can induce LLM agents to explicitly reason about balancing these cost-uncertainty tradeoffs, then act more optimally in their environments. We formalize multiple tasks, including retrieval-augmented QA and a file reading coding task, as sequential decision-making problems under uncertainty. Each problem has latent environment state that impacts the agent's performance. We introduce a framework called Calibrate-Then-Act (CTA), where we pass the agent an inferred prior about this environment state to enable it to act more optimally. This information qualitatively changes agent behavior, and adds environment sensitivity to the agent which is not learned via standard RL training. Our results on a synthetic task, QA, and file reading show that making cost-benefit tradeoffs explicit with CTA helps agents discover more optimal decision-making strategies.