Scientific and Creative Analogies in Pretrained Language Models

TL;DR

This paper introduces the SCAN dataset to evaluate the analogy reasoning abilities of large pretrained language models across dissimilar domains, revealing their current limitations in understanding complex analogies.

Contribution

The paper presents a new analogy dataset, SCAN, designed to test models on more realistic and complex analogical reasoning tasks across diverse domains.

Findings

State-of-the-art LMs perform poorly on SCAN analogies.

Existing datasets may overestimate models' analogy capabilities.

Challenges in analogy understanding remain significant for current models.

Abstract

This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.