Multi-task learning for natural language processing in the 2020s: where are we going?

TL;DR

This paper surveys recent advances in multi-task learning for NLP, highlighting progress, challenges, and future directions in leveraging shared models for improved language understanding.

Contribution

It provides a comprehensive overview of recent MTL developments in NLP, emphasizing unresolved challenges and potential research directions for the next decade.

Findings

MTL has gained renewed interest due to successes like BERT and NLP benchmarks.

Shared weights and component re-usability are key focus areas in recent MTL research.

Persistent challenges in MTL may unlock better language understanding and natural language interfaces.

Abstract

Multi-task learning (MTL) significantly pre-dates the deep learning era, and it has seen a resurgence in the past few years as researchers have been applying MTL to deep learning solutions for natural language tasks. While steady MTL research has always been present, there is a growing interest driven by the impressive successes published in the related fields of transfer learning and pre-training, such as BERT, and the release of new challenge problems, such as GLUE and the NLP Decathlon (decaNLP). These efforts place more focus on how weights are shared across networks, evaluate the re-usability of network components and identify use cases where MTL can significantly outperform single-task solutions. This paper strives to provide a comprehensive survey of the numerous recent MTL contributions to the field of natural language processing and provide a forum to focus efforts on the hardest unsolved problems in the next decade. While novel models that improve performance on NLP benchmarks are continually produced, lasting MTL challenges remain unsolved which could hold the key to better language understanding, knowledge discovery and natural language interfaces.