Structural Self-Supervised Objectives for Transformers

TL;DR

This paper introduces novel self-supervised objectives for Transformer pre-training, improving efficiency and downstream task performance, especially with limited labeled data, by aligning pre-training tasks with application structures.

Contribution

It proposes new token swapping-based pre-training objectives and structurally aligned tasks that enhance NLP model performance without modifying Transformer architectures.

Findings

RTS and C-RTS reduce pre-training time with comparable performance to MLM.

SLM outperforms MLM on certain tasks within the same computational budget.

Significant improvements on benchmarks like FEVER, ASNQ, WikiQA, and TREC-QA, especially with limited labeled data.

Abstract

This thesis focuses on improving the pre-training of natural language models using unsupervised raw data to make them more efficient and aligned with downstream applications. In the first part, we introduce three alternative pre-training objectives to BERT's Masked Language Modeling (MLM), namely Random Token Substitution (RTS), Cluster-based Random Token Substitution (C-RTS), and Swapped Language Modeling (SLM). These objectives involve token swapping instead of masking, with RTS and C-RTS aiming to predict token originality and SLM predicting the original token values. Results show that RTS and C-RTS require less pre-training time while maintaining performance comparable to MLM. Surprisingly, SLM outperforms MLM on certain tasks despite using the same computational budget. In the second part, we proposes self-supervised pre-training tasks that align structurally with downstream applications, reducing the need for labeled data. We use large corpora like Wikipedia and CC-News to train models to recognize if text spans originate from the same paragraph or document in several ways. By doing continuous pre-training, starting from existing models like RoBERTa, ELECTRA, DeBERTa, BART, and T5, we demonstrate significant performance improvements in tasks like Fact Verification, Answer Sentence Selection, and Summarization. These improvements are especially pronounced when limited annotation data is available. The proposed objectives also achieve state-of-the-art results on various benchmark datasets, including FEVER (dev set), ASNQ, WikiQA, and TREC-QA, as well as enhancing the quality of summaries. Importantly, these techniques can be easily integrated with other methods without altering the internal structure of Transformer models, making them versatile for various NLP applications.