A Fully Attention-Based Information Retriever

TL;DR

This paper introduces FABIR, a fully attention-based neural network for question-answering that achieves competitive results on SQuAD with fewer parameters and faster processing compared to traditional RNN-based models.

Contribution

The paper presents a novel fully attention-based architecture for information retrieval, replacing recurrent networks with parallelizable attention mechanisms.

Findings

FABIR achieves competitive SQuAD scores.

FABIR has fewer parameters than RNN-based models.

FABIR is faster in training and inference.

Abstract

Recurrent neural networks are now the state-of-the-art in natural language processing because they can build rich contextual representations and process texts of arbitrary length. However, recent developments on attention mechanisms have equipped feedforward networks with similar capabilities, hence enabling faster computations due to the increase in the number of operations that can be parallelized. We explore this new type of architecture in the domain of question-answering and propose a novel approach that we call Fully Attention Based Information Retriever (FABIR). We show that FABIR achieves competitive results in the Stanford Question Answering Dataset (SQuAD) while having fewer parameters and being faster at both learning and inference than rival methods.