KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering

TL;DR

KS-LLM introduces a knowledge selection method that uses triples to identify relevant evidence snippets from documents, improving large language models' performance on knowledge-intensive question answering tasks.

Contribution

The paper proposes a novel triple-based knowledge selection approach for evidence documents, enhancing LLMs' ability to utilize relevant information and reduce noise in knowledge-intensive tasks.

Findings

KS-LLM outperforms baseline methods on TriviaQA, WebQ, and NQ datasets.

Triple-based selection improves answer accuracy in question answering.

Method effectively filters relevant knowledge, boosting LLM performance.

Abstract

Large language models (LLMs) suffer from the hallucination problem and face significant challenges when applied to knowledge-intensive tasks. A promising approach is to leverage evidence documents as extra supporting knowledge, which can be obtained through retrieval or generation. However, existing methods directly leverage the entire contents of the evidence document, which may introduce noise information and impair the performance of large language models. To tackle this problem, we propose a novel Knowledge Selection of Large Language Models (KS-LLM) method, aiming to identify valuable information from evidence documents. The KS-LLM approach utilizes triples to effectively select knowledge snippets from evidence documents that are beneficial to answering questions. Specifically, we first generate triples based on the input question, then select the evidence sentences most similar to triples from the evidence document, and finally combine the evidence sentences and triples to assist large language models in generating answers. Experimental comparisons on several question answering datasets, such as TriviaQA, WebQ, and NQ, demonstrate that the proposed method surpasses the baselines and achieves the best results.