Sari Sandbox: A Virtual Retail Store Environment for Embodied AI Agents

TL;DR

Sari Sandbox is a high-fidelity virtual retail environment designed to benchmark embodied AI agents in shopping tasks, supporting VR and vision-language models, with a new dataset of human demonstrations for performance comparison.

Contribution

It introduces a comprehensive, photorealistic retail simulation environment and a dataset of human demonstrations, enabling advanced benchmarking of embodied AI in retail scenarios.

Findings

Embodied agents can navigate and manipulate retail items in the environment.

Benchmarks show performance gaps between AI agents and humans.

The environment supports scalable and realistic retail task simulations.

Abstract

We present Sari Sandbox, a high-fidelity, photorealistic 3D retail store simulation for benchmarking embodied agents against human performance in shopping tasks. Addressing a gap in retail-specific sim environments for embodied agent training, Sari Sandbox features over 250 interactive grocery items across three store configurations, controlled via an API. It supports both virtual reality (VR) for human interaction and a vision language model (VLM)-powered embodied agent. We also introduce SariBench, a dataset of annotated human demonstrations across varied task difficulties. Our sandbox enables embodied agents to navigate, inspect, and manipulate retail items, providing baselines against human performance. We conclude with benchmarks, performance analysis, and recommendations for enhancing realism and scalability. The source code can be accessed via https://github.com/upeee/sari-sandbox-env.