MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

TL;DR

This paper introduces MM-tau-p$^2$, a comprehensive benchmark with 12 metrics for evaluating the robustness of multi-modal, persona-adaptive agents in dual-control settings, considering user input and evolving behaviors.

Contribution

It presents a novel evaluation framework for multi-modal agents that incorporates persona adaptation and dual-control metrics, extending prior work with new metrics and domain-specific assessments.

Findings

State-of-the-art LLMs like GPT-5 and GPT 4.1 show additional robustness challenges.

The benchmark provides a holistic, automated evaluation method for multi-modal agents.

Domain-specific estimates demonstrate the framework's applicability in telecom and retail.

Abstract

Current evaluation frameworks and benchmarks for LLM powered agents focus on text chat driven agents, these frameworks do not expose the persona of user to the agent, thus operating in a user agnostic environment. Importantly, in customer experience management domain, the agent's behaviour evolves as the agent learns about user personality. With proliferation of real time TTS and multi-modal language models, LLM based agents are gradually going to become multi-modal. Towards this, we propose the MM-tau-p$^2$ benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query. In particular, our work shows that even with state of-the-art frontier LLMs like GPT-5, GPT 4.1, there are additional considerations measured using metrics viz. multi-modal robustness, turn overhead while introducing multi-modality into LLM based agents. Overall, MM-tau-p$^2$ builds on our prior work FOCAL and provides a holistic way of evaluating multi-modal agents in an automated way by introducing 12 novel metrics. We also provide estimates of these metrics on the telecom and retail domains by using the LLM-as-judge approach using carefully crafted prompts with well defined rubrics for evaluating each conversation.