A Learn-Then-Reason Model Towards Generalization in Knowledge Base Question Answering

TL;DR

This paper introduces KBLLaMA, a learn-then-reason approach that enhances knowledge base question answering by integrating new knowledge into large language models, significantly improving generalization across benchmarks.

Contribution

The paper proposes a novel learn-then-reason framework for KBQA that effectively incorporates new knowledge into language models, addressing knowledge boundary limitations of previous methods.

Findings

Achieves state-of-the-art results on GrailQA and Bio-chemical benchmarks.

Improves performance by up to 3.8% and 9.8% over baselines.

Demonstrates effective knowledge integration enhances generalization.

Abstract

Large-scale knowledge bases (KBs) like Freebase and Wikidata house millions of structured knowledge. Knowledge Base Question Answering (KBQA) provides a user-friendly way to access these valuable KBs via asking natural language questions. In order to improve the generalization capabilities of KBQA models, extensive research has embraced a retrieve-then-reason framework to retrieve relevant evidence for logical expression generation. These multi-stage efforts prioritize acquiring external sources but overlook the incorporation of new knowledge into their model parameters. In effect, even advanced language models and retrievers have knowledge boundaries, thereby limiting the generalization capabilities of previous KBQA models. Therefore, this paper develops KBLLaMA, which follows a learn-then-reason framework to inject new KB knowledge into a large language model for flexible end-to-end KBQA. At the core of KBLLaMA, we study (1) how to organize new knowledge about KBQA and (2) how to facilitate the learning of the organized knowledge. Extensive experiments on various KBQA generalization tasks showcase the state-of-the-art performance of KBLLaMA. Especially on the general benchmark GrailQA and domain-specific benchmark Bio-chemical, KBLLaMA respectively derives a performance gain of up to 3.8% and 9.8% compared to the baselines.