Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability

TL;DR

Pre-trained models like BERT, originally designed for text, can effectively transfer to non-text token sequence classification tasks, showing faster convergence and better performance than random models.

Contribution

This study demonstrates the transferability of text pre-trained models to non-text domains, revealing shared representations and transfer benefits beyond language tasks.

Findings

Pre-trained models outperform random models on non-text data.

Models pre-trained on text converge faster on non-text tasks.

Text and non-text pre-trained models share similar representations.

Abstract

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.