Unsupervised Natural Question Answering with a Small Model

TL;DR

This paper presents an unsupervised approach enabling small language models to answer factoid questions by leveraging external knowledge, reducing the need for extensive training on large models.

Contribution

It introduces an architecture that allows small models to answer questions using external knowledge, relying solely on unsupervised learning techniques.

Findings

Small models can answer factoid questions effectively.

External knowledge integration enhances small model capabilities.

Method reduces reliance on large, memory-intensive models.

Abstract

The recent (2019-02) demonstration of the power of huge language models such as GPT-2 to memorise the answers to factoid questions raises questions about the extent to which knowledge is being embedded directly within these large models. This short paper describes an architecture through which much smaller models can also answer such questions - by making use of 'raw' external knowledge. The contribution of this work is that the methods presented here rely on unsupervised learning techniques, complementing the unsupervised training of the Language Model. The goal of this line of research is to be able to add knowledge explicitly, without extensive training.