Gaussian Attention Model and Its Application to Knowledge Base Embedding and Question Answering

TL;DR

This paper introduces a Gaussian attention model that enhances neural memory access by allowing flexible focus control, and applies it to knowledge base embedding and question answering, effectively handling uncertainty and conjunctions.

Contribution

The paper presents a novel Gaussian attention mechanism that improves knowledge base embedding and question answering by modeling semantic distances and uncertainty.

Findings

Effective handling of path and conjunctive queries in knowledge bases

Improved question answering accuracy on FIFA World Cup dataset

Flexible attention focus from sharp to broad

Abstract

We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in the latent space reflects some notion of semantics. We use the proposed attention model as a scoring function for the embedding of a knowledge base into a continuous vector space and then train a model that performs question answering about the entities in the knowledge base. The proposed attention model can handle both the propagation of uncertainty when following a series of relations and also the conjunction of conditions in a natural way. On a dataset of soccer players who participated in the FIFA World Cup 2014, we demonstrate that our model can handle both path queries and conjunctive queries well.