ConMeC: A Dataset for Metonymy Resolution with Common Nouns

TL;DR

This paper introduces ConMeC, a new dataset of 6,000 sentences for metonymy involving common nouns, and evaluates large language models and BERT on this challenging task, highlighting their strengths and limitations.

Contribution

The paper creates the first large-scale dataset for common noun metonymy resolution and proposes a chain-of-thought prompting method for LLMs to detect metonymy.

Findings

LLMs perform comparably to BERT on clear-cut cases.

LLMs struggle with nuanced semantic instances.

The dataset is publicly available for further research.

Abstract

Metonymy plays an important role in our daily communication. People naturally think about things using their most salient properties or commonly related concepts. For example, by saying "The bus decided to skip our stop today," we actually mean that the bus driver made the decision, not the bus. Prior work on metonymy resolution has mainly focused on named entities. However, metonymy involving common nouns (such as desk, baby, and school) is also a frequent and challenging phenomenon. We argue that NLP systems should be capable of identifying the metonymic use of common nouns in context. We create a new metonymy dataset ConMeC, which consists of 6,000 sentences, where each sentence is paired with a target common noun and annotated by humans to indicate whether that common noun is used metonymically or not in that context. We also introduce a chain-of-thought based prompting method for detecting metonymy using large language models (LLMs). We evaluate our LLM-based pipeline, as well as a supervised BERT model on our dataset and three other metonymy datasets. Our experimental results demonstrate that LLMs could achieve performance comparable to the supervised BERT model on well-defined metonymy categories, while still struggling with instances requiring nuanced semantic understanding. Our dataset is publicly available at: https://github.com/SaptGhosh/ConMeC.