SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration

TL;DR

SigmaCollab is a new dataset capturing multimodal interactions in physically situated human-AI collaboration, enabling research on mixed-reality assistive tasks with real-world relevance.

Contribution

The paper introduces SigmaCollab, a multimodal dataset of human-AI collaboration in physical tasks, addressing a gap in realistic, application-driven datasets for this domain.

Findings

Provides rich multimodal data streams including audio, video, depth, and tracking.

Highlights new research challenges in physically situated human-AI collaboration.

Lays groundwork for future benchmarks in mixed-reality assistive scenarios.

Abstract

We introduce SigmaCollab, a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, multimodal data streams, such as the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. While the dataset is relatively small in size (~ 14 hours), its application-driven and interactive nature brings to the fore novel research challenges for human-AI collaboration, and provides more realistic testing grounds for various AI models operating in this space. In future work, we plan to use the dataset to construct a set of benchmarks for physically situated collaboration in mixed-reality task assistive scenarios. SigmaCollab is available at https://github.com/microsoft/SigmaCollab.