Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?

TL;DR

This paper investigates whether modern text-to-text multi-task NLP models experience task conflicts similar to traditional models, finding that conflict levels remain surprisingly consistent across different architectures.

Contribution

The study provides empirical evidence that task conflict and negative transfer are consistent across traditional and text-to-text multi-task models, challenging assumptions about architecture-specific conflicts.

Findings

Task conflict levels are similar across architectures.

Negative transfer remains constant despite architectural changes.

Text-to-text models do not necessarily reduce task conflicts.

Abstract

Traditional multi-task learning architectures train a single model across multiple tasks through a shared encoder followed by task-specific decoders. Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer. A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks. Does this new architecture suffer from task-conflicts that require specialized training algorithms? We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.