Hey AI, Can You Solve Complex Tasks by Talking to Agents?

TL;DR

This paper introduces a new benchmark, CommaQA, for training models to solve complex reasoning tasks by communicating with existing QA agents, highlighting current challenges and potential directions for future research.

Contribution

The paper presents CommaQA, a synthetic benchmark for complex reasoning via agent communication, and demonstrates the difficulty of learning this task without auxiliary supervision.

Findings

Black-box models perform poorly without supervision.

Models with gold decomposition supervision achieve perfect accuracy.

Learning to communicate with agents without supervision remains challenging.

Abstract

Training giant models from scratch for each complex task is resource- and data-inefficient. To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language. We design a synthetic benchmark, CommaQA, with three complex reasoning tasks (explicit, implicit, numeric) designed to be solved by communicating with existing QA agents. For instance, using text and table QA agents to answer questions such as "Who had the longest javelin throw from USA?". We show that black-box models struggle to learn this task from scratch (accuracy under 50\%) even with access to each agent's knowledge and gold facts supervision. In contrast, models that learn to communicate with agents outperform black-box models, reaching scores of 100\% when given gold decomposition supervision. However, we show that the challenge of learning to solve complex tasks by communicating with existing agents \emph{without relying on any auxiliary supervision or data} still remains highly elusive. We release CommaQA, along with a compositional generalization test split, to advance research in this direction. Dataset and Code available at https://github.com/allenai/commaqa.