AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

TL;DR

AdaShare introduces an adaptive, learnable sharing mechanism in multi-task neural networks that optimizes task-specific layer sharing for improved accuracy and efficiency, outperforming existing methods.

Contribution

The paper presents AdaShare, a novel method that learns task-specific sharing policies in multi-task learning, unlike fixed or handcrafted sharing schemes.

Findings

AdaShare achieves superior accuracy on benchmark datasets.

It reduces computational resources compared to traditional methods.

The learned sharing policies adapt effectively across diverse tasks.

Abstract

Multi-task learning is an open and challenging problem in computer vision. The typical way of conducting multi-task learning with deep neural networks is either through handcrafted schemes that share all initial layers and branch out at an adhoc point, or through separate task-specific networks with an additional feature sharing/fusion mechanism. Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks to achieve the best recognition accuracy, while taking resource efficiency into account. Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network. We efficiently optimize the task-specific policy jointly with the network weights, using standard back-propagation. Experiments on several challenging and diverse benchmark datasets with a variable number of tasks well demonstrate the efficacy of our approach over state-of-the-art methods. Project page: https://cs-people.bu.edu/sunxm/AdaShare/project.html.