AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling Agents

TL;DR

This paper introduces AWARE-US, a benchmark and methods for preference-aware infeasibility resolution in tool-calling agents, enabling more accurate and user-aligned query repairs in dialogue systems.

Contribution

It proposes three LLM-based methods for inferring constraint importance and introduces AWARE-US, a new benchmark for persona-grounded infeasibility resolution in conversational agents.

Findings

Local weighting best aligns with user preferences (48%)

Global weighting achieves highest correct-relaxation accuracy (56%)

All three methods outperform prior baselines in infeasibility resolution

Abstract

Tool-calling conversational agents querying structured databases often face two linked failures: underspecification (missing constraints needed for a precise query) andinfeasibility (a fully specified query returns anemptyset). Prior systems often respond with "no results" or apply ad hoc relaxations, which can violate user intent by discarding highly valued requirements. Wecast infeasibility handling as preference-aware query repair: when a query is unsatisfiable, the agent should relax the least important constraints. We propose three LLM-based methods to infer relative constraint importance from dialogue: (1) local weighting, (2) global one-shot weighting, and (3) pairwise ranking. Across extensive experiments in car recommendation, the local-weighting method trained with supervised fine-tuning and direct preference optimization best aligns with user preferences (48%), while global weighting achieves the highest correct-relaxation accuracy (56%); all three outperform prior infeasibility-resolution basel. We also introduce AWARE-US, a benchmark of 120+ persona-grounded queries requiring agents to (i) disambiguate a base request via conversa tion and (ii) resolve infeasibility in a way consistent with persona-implied preferences. For code refer to Github: https://github.com/mhtkrmz/Infeasible-task and the dataset is available on Hugging Face