Cross-Thought for Sentence Encoder Pre-training

TL;DR

Cross-Thought introduces a novel Transformer-based pre-training method for sequence encoders that improves NLP task performance by focusing on short sequences and selecting useful information for masked word prediction.

Contribution

The paper presents Cross-Thought, a new pre-training approach that enhances sequence encoder effectiveness by training on short sequences instead of full sentences.

Findings

Outperforms state-of-the-art encoders on question answering and textual entailment.

Achieves new state-of-the-art on HotpotQA full-wiki setting.

Improves intermediate information retrieval performance.

Abstract

In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering. Instead of using the original signals of full sentences, we train a Transformer-based sequence encoder over a large set of short sequences, which allows the model to automatically select the most useful information for predicting masked words. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders trained with continuous sentence signals as well as traditional masked language modeling baselines. Our proposed approach also achieves new state of the art on HotpotQA (full-wiki setting) by improving intermediate information retrieval performance.