Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

TL;DR

This paper presents a comprehensive framework for evaluating and optimizing multi-agent conversational shopping assistants, addressing key challenges in deployment and performance enhancement.

Contribution

It introduces a multi-faceted evaluation rubric, a calibrated LLM-based judging pipeline, and two novel prompt-optimization strategies for multi-agent systems.

Findings

Developed a structured evaluation rubric for conversational shopping assistants.

Created a calibrated LLM-as-judge pipeline aligned with human annotations.

Proposed two prompt-optimization strategies: Sub-agent GEPA and MAMuT GEPA.

Abstract

Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly coupled multi-agent systems. Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly preference-sensitive, and constrained by factors such as budget and inventory. In this paper, we present a practical blueprint for evaluating and optimizing conversational shopping assistants, illustrated through a production-scale AI grocery assistant. We introduce a multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations. Building on this evaluation foundation, we investigate two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA (Shao et al., 2025): (1) Sub-agent GEPA, which optimizes individual agent nodes against localized rubrics, and (2) MAMuT (Multi-Agent Multi-Turn) GEPA (Herrera et al., 2026), a novel system-level approach that jointly optimizes prompts across agents using multi-turn simulation and trajectory-level scoring. We release rubric templates and evaluation design guidance to support practitioners building production CSAs.