Is "My Favorite New Movie" My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

TL;DR

This paper introduces the Recursive Noun Phrase Challenge dataset to evaluate whether language models understand recursive noun phrases, revealing current models' limitations and potential for learning this knowledge.

Contribution

The paper presents RNPC, a novel dataset for testing recursive NP understanding, and demonstrates that models can learn this knowledge with proper training.

Findings

State-of-the-art models perform at chance on RNPC

Models can learn recursive NP understanding with appropriate data

Training on RNPC improves zero-shot performance on harm detection

Abstract

Recursive noun phrases (NPs) have interesting semantic properties. For example, "my favorite new movie" is not necessarily my favorite movie, whereas "my new favorite movie" is. This is common sense to humans, yet it is unknown whether language models have such knowledge. We introduce the Recursive Noun Phrase Challenge (RNPC), a dataset of three textual inference tasks involving textual entailment and event plausibility comparison, precisely targeting the understanding of recursive NPs. When evaluated on RNPC, state-of-the-art Transformer models only perform around chance. Still, we show that such knowledge is learnable with appropriate data. We further probe the models for relevant linguistic features that can be learned from our tasks, including modifier semantic category and modifier scope. Finally, models trained on RNPC achieve strong zero-shot performance on an extrinsic Harm Detection evaluation task, showing the usefulness of the understanding of recursive NPs in downstream applications.