SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing

TL;DR

SPAGHETTI is a hybrid open-domain question-answering system that combines semantic parsing and retrieval from diverse data sources, achieving state-of-the-art results on a comprehensive dataset.

Contribution

It introduces a novel hybrid QA pipeline that integrates semantic parsing with retrieval from heterogeneous sources, enhancing accuracy over previous methods.

Findings

Achieves 56.5% EM on Compmix dataset

Manual analysis shows over 90% accuracy

EM may not be suitable for current QA system evaluation

Abstract

We introduce SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes, a hybrid question-answering (QA) pipeline that utilizes information from heterogeneous knowledge sources, including knowledge base, text, tables, and infoboxes. Our LLM-augmented approach achieves state-of-the-art performance on the Compmix dataset, the most comprehensive heterogeneous open-domain QA dataset, with 56.5% exact match (EM) rate. More importantly, manual analysis on a sample of the dataset suggests that SPAGHETTI is more than 90% accurate, indicating that EM is no longer suitable for assessing the capabilities of QA systems today.