Improving the State of the Art for Training Human-AI Teams: Technical Report #2 -- Results of Researcher Knowledge Elicitation Survey

TL;DR

This report surveys researchers to identify key features and priorities for developing a Synthetic Task Environment (STE) to support training and research on Human-AI teaming, highlighting existing tools and desired capabilities.

Contribution

It provides insights into researcher priorities for STE development, emphasizing the need for flexible, open-source tools with automated analysis features for Human-AI team training research.

Findings

Many researchers believe suitable open-source STEs already exist.

Automated transcription and coding tools are highly valued.

Flexibility and robust data export are important features.

Abstract

A consensus report produced for the Air Force Research Laboratory (AFRL) by the National Academies of Sciences, Engineering, and Mathematics documented a prevalent and increasing desire to support human-Artificial Intelligence (AI) teaming across military service branches. Sonalysts has begun an internal initiative to explore the training of Human-AI teams. The first step in this effort is to develop a Synthetic Task Environment (STE) that is capable of facilitating research on Human-AI teams. Our goal is to create a STE that offers a task environment that could support the breadth of research that stakeholders plan to perform within this domain. As a result, we wanted to sample the priorities of the relevant research community broadly, and the effort documented in this report is our initial attempt to do so. We created a survey that featured two types of questions. The first asked respondents to report their agreement with STE features that we anticipated might be important. The second represented open-ended questions that asked respondents to specify their priorities within several dimensions of the anticipated STE. The research team invited nineteen researchers from academic and Government labs to participate, and 11 were able to complete the survey. The team analyzed their responses to identify themes that emerged and topics that would benefit from further analysis. The most significant finding of the survey was that a number of researchers felt that various open-source STEs that would meet our needs already exist. Researchers also emphasized the need for automated transcription and coding tools to ease the burden of assessing inter-team communications; the importance of robust data capture and export capabilities; and the desirability of extensive flexibility across many aspects of the tool.