Data Annealing for Informal Language Understanding Tasks

TL;DR

This paper introduces a data annealing transfer learning method that gradually incorporates informal language data into training, significantly improving model performance on informal language understanding tasks.

Contribution

It proposes a novel, model-independent data annealing approach that effectively bridges the performance gap for informal language understanding tasks using pre-trained models.

Findings

Outperforms state-of-the-art models on informal language tasks

Effective across various tasks and models

Gradually increasing informal data improves performance

Abstract

There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We pro-pose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.