A Comparative Study of Word Embeddings for Reading Comprehension

TL;DR

This paper demonstrates that the selection and handling of pre-trained word embeddings and out-of-vocabulary tokens significantly influence reading comprehension performance, often more than model architecture choices.

Contribution

It systematically evaluates the impact of embedding choices and out-of-vocabulary handling, providing practical recommendations for future research.

Findings

Pre-trained embeddings greatly affect performance.

Handling out-of-vocabulary tokens is crucial.

Minor choices can outweigh architectural innovations.

Abstract

The focus of past machine learning research for Reading Comprehension tasks has been primarily on the design of novel deep learning architectures. Here we show that seemingly minor choices made on (1) the use of pre-trained word embeddings, and (2) the representation of out-of-vocabulary tokens at test time, can turn out to have a larger impact than architectural choices on the final performance. We systematically explore several options for these choices, and provide recommendations to researchers working in this area.