SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

TL;DR

SIGMA is an open-source mixed-reality platform that combines sensing, rendering, and AI models to facilitate research on task assistance, enabling step-by-step guidance in real-world scenarios.

Contribution

The paper introduces SIGMA, an extensible open-source system that integrates mixed reality with large language and vision models for task assistance research.

Findings

Provides a flexible platform for mixed-reality task assistance research

Enables community-driven evaluation of multimodal AI models

Facilitates real-world interactive application development

Abstract

We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. We present the system's core capabilities, discuss its overall design and implementation, and outline directions for future research enabled by the system. SIGMA is easily extensible and provides a useful basis for future research at the intersection of mixed reality and AI. By open-sourcing an end-to-end implementation, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.