Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?

TL;DR

This paper investigates how noise-induced corruption of subword segmentation impacts the semantic understanding of pretrained language models, revealing their vulnerabilities to various types of segmentation disruptions.

Contribution

It introduces the CoLeS evaluation framework to systematically analyze segmentation corruption effects on PLMs' semantic comprehension under noisy conditions.

Findings

PLMs struggle with completely different subwords caused by noise.

Small subword fragments significantly impair PLMs' understanding.

Insertion of many subwords within others reduces semantic accuracy.

Abstract

For Pretrained Language Models (PLMs), their susceptibility to noise has recently been linked to subword segmentation. However, it is unclear which aspects of segmentation affect their understanding. This study assesses the robustness of PLMs against various disrupted segmentation caused by noise. An evaluation framework for subword segmentation, named Contrastive Lexical Semantic (CoLeS) probe, is proposed. It provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs. Experimental results indicate that PLMs are unable to accurately compute word meanings if the noise introduces completely different subwords, small subword fragments, or a large number of additional subwords, particularly when they are inserted within other subwords.