PASemiQA: Plan-Assisted Agent for Question Answering on Semi-Structured Data with Text and Relational Information

TL;DR

PASemiQA is a novel method that combines plan generation and LLM-based traversal to improve question answering accuracy on semi-structured data containing both text and relational information.

Contribution

It introduces a plan-assisted approach that jointly leverages text and relational data, addressing limitations of existing retrieval-augmented methods.

Findings

Effective across multiple semi-structured datasets

Improves accuracy of question answering systems

Demonstrates potential for real-world applications

Abstract

Large language models (LLMs) have shown impressive abilities in answering questions across various domains, but they often encounter hallucination issues on questions that require professional and up-to-date knowledge. To address this limitation, retrieval-augmented generation (RAG) techniques have been proposed, which retrieve relevant information from external sources to inform their responses. However, existing RAG methods typically focus on a single type of external data, such as vectorized text database or knowledge graphs, and cannot well handle real-world questions on semi-structured data containing both text and relational information. To bridge this gap, we introduce PASemiQA, a novel approach that jointly leverages text and relational information in semi-structured data to answer questions. PASemiQA first generates a plan to identify relevant text and relational information to answer the question in semi-structured data, and then uses an LLM agent to traverse the semi-structured data and extract necessary information. Our empirical results demonstrate the effectiveness of PASemiQA across different semi-structured datasets from various domains, showcasing its potential to improve the accuracy and reliability of question answering systems on semi-structured data.