CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks

TL;DR

This paper introduces CPS-TaskForge, a flexible environment generator for collaborative problem solving research, enabling the study of larger teams and diverse communication tasks through configurable game-based scenarios.

Contribution

The paper presents CPS-TaskForge, a novel CPS environment generator and a design checklist to facilitate multi-agent CPS research and data collection.

Findings

Validated the generation of diverse natural language communication in CPS tasks.

Demonstrated the use of CPS-TaskForge for creating multi-agent problem solving environments.

Discussed future research opportunities with configurable CPS tasks.

Abstract

Teams can outperform individuals; could adding AI teammates further bolster performance of teams solving problems collaboratively? Collaborative problem solving (CPS) research commonly studies teams with two agents (human-human or human-AI), but team research literature finds that, for complex tasks, larger teams are more effective. Progress in studying collaboration with more than two agents, through textual records of team interactions, is hindered by a major data challenge: available CPS corpora are predominantly dyadic, and adapting pre-existing CPS tasks to more agents is non-trivial. We address this data challenge by developing a CPS task generator, CPS-TaskForge, that can produce environments for studying CPS under a wide array of conditions, and releasing a CPS task design checklist grounded in the theoretical PISA 2015 CPS framework to help facilitate the development of CPS corpora with more agents. CPS-TaskForge takes the form of a resource management (tower defense) game, and different CPS tasks can be studied by manipulating game design parameters. We conduct a case study with groups of 3-4 humans to validate production of diverse natural language CPS communication in a game instance produced by CPS-TaskForge. We discuss opportunities for advancing research in CPS (both with human-only and human-AI teams) using different task configurations. We will release data and code.