One-Shot Learning for Language Modelling

TL;DR

This paper investigates one-shot learning in NLP by adapting matching networks, comparing various similarity measures, and establishing a benchmark for few-shot word prediction tasks using the WikiText-2 dataset.

Contribution

It explores the effectiveness of different distance metrics in few-shot learning and establishes a publicly available benchmark for future research in NLP.

Findings

No single best distance metric for k-shot learning.

Performance depends on the number of shots during training.

Benchmark dataset for one to three-shot learning in NLP is provided.

Abstract

Humans can infer a great deal about the meaning of a word, using the syntax and semantics of surrounding words even if it is their first time reading or hearing it. We can also generalise the learned concept of the word to new tasks. Despite great progress in achieving human-level performance in certain tasks (Silver et al., 2016), learning from one or few examples remains a key challenge in machine learning, and has not thoroughly been explored in Natural Language Processing (NLP). In this work we tackle the problem of oneshot learning for an NLP task by employing ideas from recent developments in machine learning: embeddings, attention mechanisms (softmax) and similarity measures (cosine, Euclidean, Poincare, and Minkowski). We adapt the framework suggested in matching networks (Vinyals et al., 2016), and explore the effectiveness of the aforementioned methods in one, two and three-shot learning problems on the task of predicting missing word explored in (Vinyals et al., 2016) by using the WikiText-2 dataset. Our work contributes in two ways: Our first contribution is that we explore the effectiveness of different distance metrics on k-shot learning, and show that there is no single best distance metric for k-shot learning, which challenges common belief. We found that the performance of a distance metric depends on the number of shots used during training. The second contribution of our work is that we establish a benchmark for one, two, and three-shot learning on a language task with a publicly available dataset that can be used to benchmark against in future research.