Multi-Task Deep Neural Networks for Natural Language Understanding

TL;DR

This paper introduces MT-DNN, a multi-task deep learning model that combines BERT with multi-task training to achieve state-of-the-art results on various NLU benchmarks and improve domain adaptation.

Contribution

The paper presents a novel multi-task learning framework that enhances BERT with shared representations, leading to improved performance and domain adaptation in NLU tasks.

Findings

Achieved new state-of-the-art on ten NLU tasks.

Improved GLUE benchmark score to 82.7%.

Enabled domain adaptation with fewer labels.

Abstract

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.