The Dark Side of the Language: Pre-trained Transformers in the DarkNet

TL;DR

This paper investigates the performance of pre-trained Transformers on unseen DarkNet sentences, revealing that domain adaptation is crucial for their success, while neural networks perform comparably without extensive pre-training.

Contribution

It demonstrates that pre-trained Transformers require extreme domain adaptation to excel on DarkNet data, and neural networks perform similarly without large pre-training datasets.

Findings

Neural networks perform on par with Transformers without extensive pre-training.

Transformers need domain-specific retraining to achieve high performance.

Pre-training corpora provide unexpected benefits for Transformers.

Abstract

Pre-trained Transformers are challenging human performances in many NLP tasks. The massive datasets used for pre-training seem to be the key to their success on existing tasks. In this paper, we explore how a range of pre-trained Natural Language Understanding models perform on definitely unseen sentences provided by classification tasks over a DarkNet corpus. Surprisingly, results show that syntactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning. Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results. This suggests that huge pre-training corpora may give Transformers unexpected help since they are exposed to many of the possible sentences.