Extreme Multi-Domain, Multi-Task Learning With Unified Text-to-Text Transfer Transformers

TL;DR

This paper explores multi-domain, multi-task learning with text-to-text transformers across Python code and Chess, revealing that GPT-style joint pretraining and finetuning strategies effectively balance task performance and domain knowledge retention.

Contribution

It introduces a comprehensive study of multi-domain, multi-task transfer learning using MD-T5 across diverse domains, comparing training strategies and evaluating performance metrics.

Findings

GPT-style joint pretraining + joint finetuning performs best across tasks

Negative transfer and catastrophic forgetting remain challenges

Multi-domain knowledge is better preserved with GPT-style joint training

Abstract

Text-to-text transformers have shown remarkable success in the task of multi-task transfer learning, especially in natural language processing (NLP). However, while there have been several attempts to train transformers on different domains, there is usually a clear relationship between these domains, e.g.,, code summarization, where the natural language summary describes the code. There have been very few attempts to study how multi-task transfer learning works on tasks in significantly different domains. In this project, we investigated the behavior of multi-domain, multi-task learning using multi-domain text-to-text transfer transformers (MD-T5) on four tasks across two domains - Python Code and Chess. We carried out extensive experiments using three popular training strategies: Bert-style joint pretraining + successive finetuning, GPT-style joint pretraining + successive finetuning, and GPT-style joint pretraining + joint finetuning. Also, we evaluate the model on four metrics - Play Score, Eval Score, BLEU Score, and Multi-Domain Learning Score (MDLS). These metrics measure performance across the various tasks and multi-domain learning. We show that while negative knowledge transfer and catastrophic forgetting are still considerable challenges for all the models, the GPT-style joint pretraining + joint finetuning strategy showed the most promise in multi-domain, multi-task learning as it performs well across all four tasks while still keeping its multi-domain knowledge.