Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA

TL;DR

This paper introduces SKP, a structured knowledge-aware pre-training method for KBQA that improves understanding of complex subgraphs and enhances retrieval accuracy, addressing the limitations of traditional PLMs in structured KB contexts.

Contribution

The paper proposes novel pre-training tasks and strategies that enable models to better encode and reason over structured knowledge graphs in KBQA tasks.

Findings

Significant improvement in subgraph retrieval accuracy (+4.08% H@10).

Effective encoding of complex subgraphs through new linearization and attention mechanisms.

Validated on WebQSP dataset with notable performance gains.

Abstract

Knowledge Base Question Answering (KBQA) aims to answer natural language questions with factual information such as entities and relations in KBs. However, traditional Pre-trained Language Models (PLMs) are directly pre-trained on large-scale natural language corpus, which poses challenges for them in understanding and representing complex subgraphs in structured KBs. To bridge the gap between texts and structured KBs, we propose a Structured Knowledge-aware Pre-training method (SKP). In the pre-training stage, we introduce two novel structured knowledge-aware tasks, guiding the model to effectively learn the implicit relationship and better representations of complex subgraphs. In downstream KBQA task, we further design an efficient linearization strategy and an interval attention mechanism, which assist the model to better encode complex subgraphs and shield the interference of irrelevant subgraphs during reasoning respectively. Detailed experiments and analyses on WebQSP verify the effectiveness of SKP, especially the significant improvement in subgraph retrieval (+4.08% H@10).