Should I Have Expressed a Different Intent? Counterfactual Generation for LLM-Based Autonomous Control

TL;DR

This paper presents a formal framework for counterfactual reasoning in LLM-based autonomous agents, enabling users to explore alternative outcomes of their high-level intents with reliability guarantees.

Contribution

It introduces conformal counterfactual generation (CCG), a novel method that models agent interactions as a structural causal model and provides probabilistic guarantees on counterfactual outcomes.

Findings

CCG effectively generates high-probability sets of counterfactual outcomes.

Demonstrated advantages over naive re-execution baselines in wireless network control.

Provides formal reliability guarantees for counterfactual reasoning.

Abstract

Large language model (LLM)-powered agents can translate high-level user intents into plans and actions in an environment. Yet after observing an outcome, users may wonder: What if I had phrased my intent differently? We introduce a framework that enables such counterfactual reasoning in agentic LLM-driven control scenarios, while providing formal reliability guarantees. Our approach models the closed-loop interaction between a user, an LLM-based agent, and an environment as a structural causal model (SCM), and leverages test-time scaling to generate multiple candidate counterfactual outcomes via probabilistic abduction. Through an offline calibration phase, the proposed conformal counterfactual generation (CCG) yields sets of counterfactual outcomes that are guaranteed to contain the true counterfactual outcome with high probability. We showcase the performance of CCG on a wireless network control use case, demonstrating significant advantages compared to naive re-execution baselines.