Symmetric Regularization based BERT for Pair-wise Semantic Reasoning

TL;DR

This paper enhances BERT's sentence-pair reasoning by introducing a symmetric regularization with a 3-class task, including PSP, to better distinguish entailment from shallow correlation, improving performance on NLI and MRC tasks.

Contribution

It proposes a novel symmetric regularization approach with a 3-class categorization task for BERT, incorporating document-level context and label smoothing to improve semantic reasoning.

Findings

Significant performance improvements on NLI and MRC benchmarks.

Enhanced ability to distinguish entailment from shallow correlation.

Effective use of document-level information in pre-training.

Abstract

The ability of semantic reasoning over the sentence pair is essential for many natural language understanding tasks, e.g., natural language inference and machine reading comprehension. A recent significant improvement in these tasks comes from BERT. As reported, the next sentence prediction (NSP) in BERT, which learns the contextual relationship between two sentences, is of great significance for downstream problems with sentence-pair input. Despite the effectiveness of NSP, we suggest that NSP still lacks the essential signal to distinguish between entailment and shallow correlation. To remedy this, we propose to augment the NSP task to a 3-class categorization task, which includes a category for previous sentence prediction (PSP). The involvement of PSP encourages the model to focus on the informative semantics to determine the sentence order, thereby improves the ability of semantic understanding. This simple modification yields remarkable improvement against vanilla BERT. To further incorporate the document-level information, the scope of NSP and PSP is expanded into a broader range, i.e., NSP and PSP also include close but nonsuccessive sentences, the noise of which is mitigated by the label-smoothing technique. Both qualitative and quantitative experimental results demonstrate the effectiveness of the proposed method. Our method consistently improves the performance on the NLI and MRC benchmarks, including the challenging HANS dataset \cite{hans}, suggesting that the document-level task is still promising for the pre-training.