Pretrained Transformers Do not Always Improve Robustness

TL;DR

Pretrained Transformers, despite their success in Out of Distribution robustness, may be less robust than traditional models when exposed to noisy data, and adversarial filtering does not always enhance their robustness.

Contribution

This study provides an empirical comparison of PT and traditional models under noisy conditions and evaluates the effectiveness of adversarial filtering in improving robustness.

Findings

PT are less robust than traditional models on noisy data

Adversarial filtering does not always improve PT robustness

Noisy data can fool adversarial filtering mechanisms

Abstract

Pretrained Transformers (PT) have been shown to improve Out of Distribution (OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs, Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings. How does the robustness comparison hold in a real world setting where some part of the dataset can be noisy? Do PT also provide more robust representation than traditional models on exposure to noisy data? We perform a comparative study on 10 models and find an empirical evidence that PT provide less robust representation than traditional models on exposure to noisy data. We investigate further and augment PT with an adversarial filtering (AF) mechanism that has been shown to improve OOD generalization. However, increase in generalization does not necessarily increase robustness, as we find that noisy data fools the AF method powered by PT.