UserBench: An Interactive Gym Environment for User-Centric Agents

TL;DR

UserBench is a new interactive benchmark for evaluating user-centric agents, revealing current LLMs' limitations in proactive collaboration and preference understanding during multi-turn, goal-driven interactions.

Contribution

We introduce UserBench, a benchmark for assessing agents' ability to proactively clarify goals and preferences in user interactions, highlighting gaps in current LLM capabilities.

Findings

Models align with user intents only 20% of the time

Most models uncover less than 30% of user preferences

Current agents struggle with proactive, collaborative behavior

Abstract

Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.