CREW: Facilitating Human-AI Teaming Research

TL;DR

CREW is a versatile platform designed to advance Human-AI teaming research by supporting real-time decision-making, multimodal physiological data collection, and benchmarking AI agents, thereby enabling multidisciplinary studies and rapid validation.

Contribution

CREW introduces a modular, real-time Human-AI teaming platform that supports multimodal data collection and benchmarking, addressing limitations of existing research tools.

Findings

Conducted 50 human subject studies in one week

Validated effectiveness of AI benchmarks in Human-AI teaming

Supports multidisciplinary research with real-time decision-making

Abstract

With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.