Towards Collaborative Question Answering: A Preliminary Study

TL;DR

This paper introduces CollabQA, a new collaborative question answering task involving multiple expert agents working together, supported by a synthetic dataset and evaluation metrics, highlighting challenges and future extensions.

Contribution

It proposes a novel collaborative QA framework with a synthetic dataset, models, and evaluation methods, addressing the complexities of multi-expert cooperation.

Findings

Collaboration is challenging without prior structure unless experts are perfect.

Synthetic dataset effectively models distributed knowledge for collaborative QA.

Performance drops significantly without structured collaboration or perfect experts.

Abstract

Knowledge and expertise in the real-world can be disjointedly owned. To solve a complex question, collaboration among experts is often called for. In this paper, we propose CollabQA, a novel QA task in which several expert agents coordinated by a moderator work together to answer questions that cannot be answered with any single agent alone. We make a synthetic dataset of a large knowledge graph that can be distributed to experts. We define the process to form a complex question from ground truth reasoning path, neural network agent models that can learn to solve the task, and evaluation metrics to check the performance. We show that the problem can be challenging without introducing prior of the collaboration structure, unless experts are perfect and uniform. Based on this experience, we elaborate extensions needed to approach collaboration tasks in real-world settings.