Probabilistic Linguistic Knowledge and Token-level Text Augmentation

TL;DR

This study evaluates token-level text augmentation techniques and the impact of probabilistic linguistic knowledge, finding limited effectiveness of these methods across different models and languages in a question matching task.

Contribution

It introduces REDA and REDA_NG augmentation methods and provides a comprehensive evaluation of their effectiveness and the role of probabilistic linguistic knowledge.

Findings

Token-level augmentation techniques are generally ineffective.

Probabilistic linguistic knowledge has minimal impact.

Results are consistent across Chinese and English datasets.

Abstract

This paper investigates the effectiveness of token-level text augmentation and the role of probabilistic linguistic knowledge within a linguistically-motivated evaluation context. Two text augmentation programs, REDA and REDA$_{NG}$, were developed, both implementing five token-level text editing operations: Synonym Replacement (SR), Random Swap (RS), Random Insertion (RI), Random Deletion (RD), and Random Mix (RM). REDA$_{NG}$ leverages pretrained $n$-gram language models to select the most likely augmented texts from REDA's output. Comprehensive and fine-grained experiments were conducted on a binary question matching classification task in both Chinese and English. The results strongly refute the general effectiveness of the five token-level text augmentation techniques under investigation, whether applied together or separately, and irrespective of various common classification model types used, including transformers. Furthermore, the role of probabilistic linguistic knowledge is found to be minimal.