Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI

TL;DR

Pre-trained language models demonstrate strong cross-domain generalization capabilities, outperforming models trained from scratch on various tasks, and their effectiveness is maintained even with reduced parameters, advancing progress toward general AI.

Contribution

This study systematically evaluates pre-trained language models on diverse non-language tasks, highlighting their generalization ability and the importance of pre-trained embeddings.

Findings

Pre-trained models outperform from-scratch transformers by large margins.

Reducing parameters in pre-trained models has minimal performance impact.

Pre-trained embeddings are essential for optimal results.

Abstract

Pre-trained language models have recently emerged as a powerful tool for fine-tuning a variety of language tasks. Ideally, when models are pre-trained on large amount of data, they are expected to gain implicit knowledge. In this paper, we investigate the ability of pre-trained language models to generalize to different non-language tasks. In particular, we test them on tasks from different domains such as computer vision, reasoning on hierarchical data, and protein fold prediction. The four pre-trained models that we used, T5, BART, BERT, and GPT-2 achieve outstanding results. They all have similar performance and they outperform transformers that are trained from scratch by a large margin. For instance, pre-trained language models perform better on the Listops dataset, with an average accuracy of 58.7\%, compared to transformers trained from scratch, which have an average accuracy of 29.0\%. The significant improvement demonstrated across three types of datasets suggests that pre-training on language helps the models to acquire general knowledge, bringing us a step closer to general AI. We also showed that reducing the number of parameters in pre-trained language models does not have a great impact as the performance drops slightly when using T5-Small instead of T5-Base. In fact, when using only 2\% of the parameters, we achieved a great improvement compared to training from scratch. Finally, in contrast to prior work, we find out that using pre-trained embeddings for the input layer is necessary to achieve the desired results.