Improving the State of the Art for Training Human-AI Teams: Technical Report #1 -- Results of Subject-Matter Expert Knowledge Elicitation Survey

TL;DR

This report details the initial efforts to develop a Synthetic Task Environment (STE) for training human-AI teams in military command scenarios, focusing on JADC2 and incorporating expert insights.

Contribution

It introduces a framework for creating a STE tailored to military human-AI teaming, based on expert elicitation and analysis of operational communication and decision-making challenges.

Findings

SMEs favor tasks involving complex data processing for decision-making.

The STE should support 'teams of teams' in a matrixed organization.

Communication modalities should include spoken, text-based, and face-to-face interactions.

Abstract

A consensus report produced for the Air Force Research Laboratory by the National Academies of Sciences, Engineering, and Mathematics documented a prevalent and increasing desire to support human-Artificial Intelligence (AI) teaming across military service branches. Sonalysts has begun an internal initiative to explore the training of human-AI teams. The first step in this effort is to develop a Synthetic Task Environment (STE) that is capable of facilitating research on human-AI teams. We decided to use Joint All-Domain Command and Control (JADC2) as a focus point for developing the STE because the volume of sensor inputs and decision options within the JADC2 concept likely requires the use of AI systems to enable timely decisions. Given this focus, we engaged a number of Subject-Matter Experts (SMEs) with Command and Control experience to gain insight into developing a STE that embodied the teaming challenges associated with JADC2. This report documents our initial engagement with those stakeholders. The research team identified thirteen Sonalysts employees with military backgrounds and Command and Control experience, and invited them to participate. Twelve respondents completed the survey. The team then analyzed the responses to identify themes that emerged and topics that would benefit from further analysis. The results indicated that our SMEs were amenable to research using tasks that were analogous to those encountered in military environments, as long as they required teams to process a great deal of incoming data to arrive at complex decisions. The SMEs felt that the testbed should support 'teams of teams" that represented a matrixed organization, and that it should support a robust array to spoken, text-based, and face-to-face communications.