Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance

TL;DR

This paper explores whether language models can learn analogical reasoning, demonstrating that they can do so with limited data and approaching human performance on relevant tasks.

Contribution

The study introduces methods for training language models to learn analogical reasoning and compares their performance to human benchmarks.

Findings

Models can learn analogical reasoning with limited data.

Trained models approach human performance on analogy tasks.

Training objectives significantly impact learning success.

Abstract

While analogies are a common way to evaluate word embeddings in NLP, it is also of interest to investigate whether or not analogical reasoning is a task in itself that can be learned. In this paper, we test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans than those in commonly used NLP benchmarks. Our experiments find that models are able to learn analogical reasoning, even with a small amount of data. We additionally compare our models to a dataset with a human baseline, and find that after training, models approach human performance.