Transfer Fine-Tuning: A BERT Case Study

TL;DR

This paper introduces a method to enhance smaller BERT models for semantic equivalence tasks by injecting paraphrase relations, achieving performance comparable or superior to larger models without increasing size.

Contribution

The study proposes a novel approach to improve BERT's transfer learning capabilities by injecting paraphrase relations, reducing the need for larger models.

Findings

Smaller BERT models with injected paraphrase relations outperform larger models on semantic tasks.

The method yields larger gains on tasks with limited training data.

Enhanced models maintain performance while keeping the same size.

Abstract

A semantic equivalence assessment is defined as a task that assesses semantic equivalence in a sentence pair by binary judgment (i.e., paraphrase identification) or grading (i.e., semantic textual similarity measurement). It constitutes a set of tasks crucial for research on natural language understanding. Recently, BERT realized a breakthrough in sentence representation learning (Devlin et al., 2019), which is broadly transferable to various NLP tasks. While BERT's performance improves by increasing its model size, the required computational power is an obstacle preventing practical applications from adopting the technology. Herein, we propose to inject phrasal paraphrase relations into BERT in order to generate suitable representations for semantic equivalence assessment instead of increasing the model size. Experiments on standard natural language understanding tasks confirm that our method effectively improves a smaller BERT model while maintaining the model size. The generated model exhibits superior performance compared to a larger BERT model on semantic equivalence assessment tasks. Furthermore, it achieves larger performance gains on tasks with limited training datasets for fine-tuning, which is a property desirable for transfer learning.