Towards Reliable Multi-Agent Systems for Marketing Applications via Reflection, Memory, and Planning

TL;DR

This paper presents RAMP, a multi-agent framework utilizing planning, memory, and verification to enhance the reliability and accuracy of LLM-based marketing tools, demonstrating significant improvements in task performance and user satisfaction.

Contribution

It introduces RAMP, a novel multi-agent system with iterative planning, verification, and long-term memory, improving reliability and accuracy in real-world marketing applications.

Findings

Accuracy increased by 28 percentage points with RAMP.

Iterative verification improved recall by roughly 20 percentage points.

Enhanced user satisfaction through more reliable LLM-based systems.

Abstract

Recent advances in large language models (LLMs) enabled the development of AI agents that can plan and interact with tools to complete complex tasks. However, literature on their reliability in real-world applications remains limited. In this paper, we introduce a multi-agent framework for a marketing task: audience curation. To solve this, we introduce a framework called RAMP that iteratively plans, calls tools, verifies the output, and generates suggestions to improve the quality of the audience generated. Additionally, we equip the model with a long-term memory store, which is a knowledge base of client-specific facts and past queries. Overall, we demonstrate the use of LLM planning and memory, which increases accuracy by 28 percentage points on a set of 88 evaluation queries. Moreover, we show the impact of iterative verification and reflection on more ambiguous queries, showing progressively better recall (roughly +20 percentage points) with more verify/reflect iterations on a smaller challenge set, and higher user satisfaction. Our results provide practical insights for deploying reliable LLM-based systems in dynamic, industry-facing environments.