Toward Scalable and Controllable AR Experimentation

TL;DR

This paper introduces ExpAR, a scalable and controllable platform for AR experimentation that addresses evaluation challenges by enabling shared physical resources and standardized pipelines, facilitating fairer and larger-scale AR system assessments.

Contribution

The paper presents the design and preliminary prototype of ExpAR, a platform that allows scalable, controllable, and shared AR experiments to improve evaluation fairness and reproducibility.

Findings

Feasibility demonstrated with a prototype

Preliminary evaluations highlight device streaming capabilities

Platform supports standalone and federated deployment

Abstract

To understand how well a proposed augmented reality (AR) solution works, existing papers often conducted tailored and isolated evaluations for specific AR tasks, e.g., depth or lighting estimation, and compared them to easy-to-setup baselines, either using datasets or resorting to time-consuming data capturing. Conceptually simple, it can be extremely difficult to evaluate an AR system fairly and in scale to understand its real-world performance. The difficulties arise for three key reasons: lack of control of the physical environment, the time-consuming data capturing, and the difficulties to reproduce baseline results. This paper presents our design of an AR experimentation platform, ExpAR, aiming to provide scalable and controllable AR experimentation. ExpAR is envisioned to operate as a standalone deployment or a federated platform; in the latter case, AR researchers can contribute physical resources, including scene setup and capturing devices, and allow others to time share these resources. Our design centers around the generic sensing-understanding-rendering pipeline and is driven by the evaluation limitations observed in recent AR systems papers. We demonstrate the feasibility of this vision with a preliminary prototype and our preliminary evaluations suggest the importance of further investigating different device capabilities to stream in 30 FPS. The ExpAR project site can be found at https://cake.wpi.edu/expar.