Can Pre-trained Language Models Interpret Similes as Smart as Human?

TL;DR

This paper evaluates pre-trained language models' ability to interpret similes by introducing a novel probing task and enhancing models with simile knowledge, showing improved performance but still lagging behind humans.

Contribution

The paper proposes a new simile property probing task and a knowledge-enhanced training method to improve PLMs' simile interpretation capabilities.

Findings

PLMs can infer shared simile properties but underperform humans.

Knowledge embedding improves probing accuracy by 8.58%.

Enhanced models show slight improvements in sentiment classification.

Abstract

Simile interpretation is a crucial task in natural language processing. Nowadays, pre-trained language models (PLMs) have achieved state-of-the-art performance on many tasks. However, it remains under-explored whether PLMs can interpret similes or not. In this paper, we investigate the ability of PLMs in simile interpretation by designing a novel task named Simile Property Probing, i.e., to let the PLMs infer the shared properties of similes. We construct our simile property probing datasets from both general textual corpora and human-designed questions, containing 1,633 examples covering seven main categories. Our empirical study based on the constructed datasets shows that PLMs can infer similes' shared properties while still underperforming humans. To bridge the gap with human performance, we additionally design a knowledge-enhanced training objective by incorporating the simile knowledge into PLMs via knowledge embedding methods. Our method results in a gain of 8.58% in the probing task and 1.37% in the downstream task of sentiment classification. The datasets and code are publicly available at https://github.com/Abbey4799/PLMs-Interpret-Simile.