SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

TL;DR

SimGym is a framework that simulates A/B testing in e-commerce using vision-language model agents, enabling rapid, traffic-grounded evaluation of UI changes without affecting real users.

Contribution

It introduces a novel simulation framework with traffic-grounded personas and live-browser agents that accurately predict real buyer responses to UI modifications.

Findings

Achieves 77% directional alignment with real buyer behavior in A/B tests.

Reduces experimental cycle time from weeks to under an hour.

Validates simulation accuracy across diverse storefronts and product categories.

Abstract

A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.