The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders

TL;DR

This paper investigates the limitations of multi-task learning with transformer encoders across diverse NLP tasks, revealing interference among attention heads and proposing the Stem Cell Hypothesis to explain task-specific head specialization.

Contribution

It introduces the Stem Cell Hypothesis, identifying attention heads naturally suited for multiple tasks and explaining their interference in multi-task learning.

Findings

Attention heads are claimed by multiple tasks, causing interference.

Certain attention heads are naturally talented for many tasks.

Attention head transformation varies across tasks during MTL.

Abstract

Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.