Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing

TL;DR

This paper introduces a multi-modal user agent framework for A/B testing in recommender systems, aiming to reduce costs and improve simulation fidelity by mimicking real user interactions within a constructed sandbox environment.

Contribution

It presents a novel multi-modal user agent and recommendation sandbox environment that simulate complex user behaviors for more effective A/B testing in recommender systems.

Findings

The agent can effectively simulate human decision-making processes.

Generated data enhances recommendation model performance.

The framework reduces costs and improves testing realism.

Abstract

In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent.