GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks

TL;DR

GradTS is a gradient-based method for automatically selecting high-quality auxiliary tasks in multi-task learning with Transformer models, improving performance and efficiency over existing methods like AUTOSEM.

Contribution

It introduces a novel gradient calculation approach for automatic auxiliary task selection in Transformer-based models, reducing time and manual effort.

Findings

GradTS improves MT-DNN performance on GLUE tasks by up to 17.93%.

GradTS reduces selection time by approximately 21.32% compared to AUTOSEM.

GradTS demonstrates robustness across various task and model configurations.

Abstract

A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. This paper presents GradTS, an automatic auxiliary task selection method based on gradient calculation in Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradient calculations are based on single-task experiments and (2) the gradients are re-used without additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with comparable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed objectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL research without requiring manual task filtering or costly parameter tuning.