Question Answering over Knowledge Base using Language Model Embeddings

TL;DR

This paper explores using pre-trained language model embeddings, specifically BERT, with a two-way attention mechanism and CNN architecture to improve knowledge base question answering accuracy and address limitations of existing embedding methods.

Contribution

It introduces a novel approach combining BERT embeddings with a multi-head attention mechanism and CNN for enhanced knowledge base question answering.

Findings

BERT embeddings outperform other methods in KBQA tasks.

Two-way attention improves question-answer mapping accuracy.

The proposed model shows superior results in experiments.

Abstract

Knowledge Base, represents facts about the world, often in some form of subsumption ontology, rather than implicitly, embedded in procedural code, the way a conventional computer program does. While there is a rapid growth in knowledge bases, it poses a challenge of retrieving information from them. Knowledge Base Question Answering is one of the promising approaches for extracting substantial knowledge from Knowledge Bases. Unlike web search, Question Answering over a knowledge base gives accurate and concise results, provided that natural language questions can be understood and mapped precisely to an answer in the knowledge base. However, some of the existing embedding-based methods for knowledge base question answering systems ignore the subtle correlation between the question and the Knowledge Base (e.g., entity types, relation paths, and context) and suffer from the Out Of Vocabulary problem. In this paper, we focused on using a pre-trained language model for the Knowledge Base Question Answering task. Firstly, we used Bert base uncased for the initial experiments. We further fine-tuned these embeddings with a two-way attention mechanism from the knowledge base to the asked question and from the asked question to the knowledge base answer aspects. Our method is based on a simple Convolutional Neural Network architecture with a Multi-Head Attention mechanism to represent the asked question dynamically in multiple aspects. Our experimental results show the effectiveness and the superiority of the Bert pre-trained language model embeddings for question answering systems on knowledge bases over other well-known embedding methods.