ExpertQA: Expert-Curated Questions and Attributed Answers
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark, Yatskar, Dan Roth

TL;DR
ExpertQA is a high-quality dataset of 2177 domain-specific questions with verified answers and attributions, created through expert evaluation and improvement, aimed at enhancing factuality and attribution in language model responses.
Contribution
This work introduces ExpertQA, a novel dataset with expert-curated questions and verified answers across 32 fields, facilitating research on factuality and attribution in domain-specific language models.
Findings
Expert evaluation improves response quality and attribution accuracy.
The dataset covers 32 diverse fields of study.
Expert involvement ensures high factual correctness.
Abstract
As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsBusiness Process Modeling and Analysis
