Towards Collaborative Question Answering: A Preliminary Study
Xiangkun Hu, Hang Yan, Qipeng Guo, Xipeng Qiu, Weinan Zhang, Zheng, Zhang

TL;DR
This paper introduces CollabQA, a new collaborative question answering task involving multiple expert agents working together, supported by a synthetic dataset and evaluation metrics, highlighting challenges and future extensions.
Contribution
It proposes a novel collaborative QA framework with a synthetic dataset, models, and evaluation methods, addressing the complexities of multi-expert cooperation.
Findings
Collaboration is challenging without prior structure unless experts are perfect.
Synthetic dataset effectively models distributed knowledge for collaborative QA.
Performance drops significantly without structured collaboration or perfect experts.
Abstract
Knowledge and expertise in the real-world can be disjointedly owned. To solve a complex question, collaboration among experts is often called for. In this paper, we propose CollabQA, a novel QA task in which several expert agents coordinated by a moderator work together to answer questions that cannot be answered with any single agent alone. We make a synthetic dataset of a large knowledge graph that can be distributed to experts. We define the process to form a complex question from ground truth reasoning path, neural network agent models that can learn to solve the task, and evaluation metrics to check the performance. We show that the problem can be challenging without introducing prior of the collaboration structure, unless experts are perfect and uniform. Based on this experience, we elaborate extensions needed to approach collaboration tasks in real-world settings.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExpert finding and Q&A systems · Topic Modeling · Advanced Graph Neural Networks
