DMTG: One-Shot Differentiable Multi-Task Grouping

TL;DR

DMTG introduces a fully differentiable, one-shot method for multi-task grouping that efficiently identifies task groups and trains models simultaneously, outperforming sequential approaches.

Contribution

It proposes a novel differentiable pruning approach for multi-task grouping that improves efficiency and reduces bias compared to prior sequential methods.

Findings

Improves training efficiency over sequential methods.

Effectively exploits high-order task affinity.

Achieves promising performance on CelebA and Taskonomy datasets.

Abstract

We aim to address Multi-Task Learning (MTL) with a large number of tasks by Multi-Task Grouping (MTG). Given N tasks, we propose to simultaneously identify the best task groups from 2^N candidates and train the model weights simultaneously in one-shot, with the high-order task-affinity fully exploited. This is distinct from the pioneering methods which sequentially identify the groups and train the model weights, where the group identification often relies on heuristics. As a result, our method not only improves the training efficiency, but also mitigates the objective bias introduced by the sequential procedures that potentially lead to a suboptimal solution. Specifically, we formulate MTG as a fully differentiable pruning problem on an adaptive network architecture determined by an underlying Categorical distribution. To categorize N tasks into K groups (represented by K encoder branches), we initially set up KN task heads, where each branch connects to all N task heads to exploit the high-order task-affinity. Then, we gradually prune the KN heads down to N by learning a relaxed differentiable Categorical distribution, ensuring that each task is exclusively and uniquely categorized into only one branch. Extensive experiments on CelebA and Taskonomy datasets with detailed ablations show the promising performance and efficiency of our method. The codes are available at https://github.com/ethanygao/DMTG.