Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA Communicating with Computers Program

TL;DR

This paper proposes a framework with key properties and hallmarks for assessing human-machine collaboration systems engaged in open-ended, creative, and complex tasks, guiding research and evaluation.

Contribution

It introduces a novel assessment framework for open-ended human-machine collaboration, focusing on key properties and observable hallmarks to evaluate progress.

Findings

Framework applied to story and music generation

Framework used in molecular mechanism exploration in cancer

Identifies key properties and hallmarks of successful collaboration

Abstract

There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria. We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success -- capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.