A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

TL;DR

This paper presents a joint multi-task neural network that incrementally grows in depth to handle various NLP tasks simultaneously, leveraging hierarchical linguistic structures and regularization to prevent interference.

Contribution

Introduces a novel multi-task model that grows in depth and incorporates shortcut connections, enabling effective learning across multiple NLP tasks within a single unified framework.

Findings

Achieves state-of-the-art results on five NLP tasks.

Effectively models hierarchical linguistic information.

Reduces catastrophic interference among tasks.

Abstract

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.