Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

TL;DR

This paper investigates whether large pre-trained language models encode metaphorical knowledge, examining their ability to generalize across datasets and languages, and finds that they do encode such knowledge mainly in middle layers, with good transferability.

Contribution

The study provides empirical evidence that PLMs encode metaphorical knowledge in their representations, and demonstrates their cross-lingual and cross-dataset generalization capabilities.

Findings

Metaphorical knowledge is encoded mainly in middle layers of PLMs.

PLMs' metaphorical knowledge transfers well across languages and datasets.

Consistent annotation improves transferability of metaphor detection.

Abstract

Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre-trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross-dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cognitive and NLP scientists.