From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

TL;DR

This paper evaluates GPT-3's ability to interpret and conceptualize noun compounds, showing near-perfect performance on standard tasks and exploring its reasoning versus parroting behavior on novel compounds.

Contribution

It introduces modifications to the noun compound interpretation task and investigates GPT-3's creative conceptualization abilities for rare compounds.

Findings

GPT-3 nearly perfect at standard noun compound interpretation

GPT-3 outperforms humans in conceptualizing novel compounds

GPT-3's outputs often overlap with web corpus data

Abstract

Noun compound interpretation is the task of expressing a noun compound (e.g. chocolate bunny) in a free-text paraphrase that makes the relationship between the constituent nouns explicit (e.g. bunny-shaped chocolate). We propose modifications to the data and evaluation setup of the standard task (Hendrickx et al., 2013), and show that GPT-3 solves it almost perfectly. We then investigate the task of noun compound conceptualization, i.e. paraphrasing a novel or rare noun compound. E.g., chocolate crocodile is a crocodile-shaped chocolate. This task requires creativity, commonsense, and the ability to generalize knowledge about similar concepts. While GPT-3's performance is not perfect, it is better than that of humans -- likely thanks to its access to vast amounts of knowledge, and because conceptual processing is effortful for people (Connell and Lynott, 2012). Finally, we estimate the extent to which GPT-3 is reasoning about the world vs. parroting its training data. We find that the outputs from GPT-3 often have significant overlap with a large web corpus, but that the parroting strategy is less beneficial for novel noun compounds.