Improving the State of the Art for Training Human-AI Teams: Technical Report #3 -- Analysis of Testbed Alternatives

TL;DR

This paper systematically evaluates existing human-AI teaming testbeds to identify suitable options for developing a Synthetic Task Environment, aiming to enhance research in human-AI collaboration.

Contribution

It provides a comprehensive analysis and evaluation framework for existing testbeds, identifying five promising candidates for future research development.

Findings

Identified five promising testbeds for human-AI teaming research.

Developed criteria for evaluating testbeds.

Conducted qualitative and quantitative assessments of candidates.

Abstract

Sonalysts is working on an initiative to expand our current expertise in teaming to Human-Artificial Intelligence (AI) teams by developing original research in this area. To provide a foundation for that research, Sonalysts is investigating the development of a Synthetic Task Environment (STE). In a previous report, we documented the findings of a recent outreach effort in which we asked military Subject Matter Experts (SMEs) and other researchers in the Human-AI teaming domain to identify the qualities that they most valued in a testbed. A surprising finding from that outreach was that several respondents recommended that our team look into existing human-AI teaming testbeds, rather than creating something new. Based on that recommendation, we conducted a systematic investigation of the associated landscape. In this report, we describe the results of that investigation. Building on the survey results, we developed testbed evaluation criteria, identified potential testbeds, and conducted qualitative and quantitative evaluations of candidate testbeds. The evaluation process led to five candidate testbeds for the research team to consider. In the coming months, we will assess the viability of the various alternatives and begin to execute our program of research.