Boosting Share Routing for Multi-task Learning

TL;DR

This paper introduces MTNAS, a neural architecture search framework that optimizes task sharing routes in multi-task learning, leading to improved performance and reduced negative transfer across recommendation datasets.

Contribution

The paper proposes a flexible, dynamic sharing route search space for MTL, enabling automatic discovery of effective task sharing architectures beyond traditional parameter sharing methods.

Findings

MTNAS outperforms single-task and typical MTL models on recommendation datasets.

It learns sparse sharing routes that mitigate negative transfer.

The approach maintains high computational efficiency.

Abstract

Multi-task learning (MTL) aims to make full use of the knowledge contained in multi-task supervision signals to improve the overall performance. How to make the knowledge of multiple tasks shared appropriately is an open problem for MTL. Most existing deep MTL models are based on parameter sharing. However, suitable sharing mechanism is hard to design as the relationship among tasks is complicated. In this paper, we propose a general framework called Multi-Task Neural Architecture Search (MTNAS) to efficiently find a suitable sharing route for a given MTL problem. MTNAS modularizes the sharing part into multiple layers of sub-networks. It allows sparse connection among these sub-networks and soft sharing based on gating is enabled for a certain route. Benefiting from such setting, each candidate architecture in our search space defines a dynamic sparse sharing route which is more flexible compared with full-sharing in previous approaches. We show that existing typical sharing approaches are sub-graphs in our search space. Extensive experiments on three real-world recommendation datasets demonstrate MTANS achieves consistent improvement compared with single-task models and typical multi-task methods while maintaining high computation efficiency. Furthermore, in-depth experiments demonstrates that MTNAS can learn suitable sparse route to mitigate negative transfer.