Curbing Task Interference using Representation Similarity-Guided Multi-Task Feature Sharing

TL;DR

This paper introduces Progressive Decoder Fusion, a method that uses representation similarity to effectively share task decoders in multi-task learning, improving generalization, robustness, and task consistency.

Contribution

It proposes a novel decoder fusion approach guided by representation similarity to reduce task interference in multi-task learning.

Findings

Enhanced generalization to in-distribution and out-of-distribution data.

Improved robustness against adversarial attacks.

More consistent task predictions within the multi-task network.

Abstract

Multi-task learning of dense prediction tasks, by sharing both the encoder and decoder, as opposed to sharing only the encoder, provides an attractive front to increase both accuracy and computational efficiency. When the tasks are similar, sharing the decoder serves as an additional inductive bias providing more room for tasks to share complementary information among themselves. However, increased sharing exposes more parameters to task interference which likely hinders both generalization and robustness. Effective ways to curb this interference while exploiting the inductive bias of sharing the decoder remains an open challenge. To address this challenge, we propose Progressive Decoder Fusion (PDF) to progressively combine task decoders based on inter-task representation similarity. We show that this procedure leads to a multi-task network with better generalization to in-distribution and out-of-distribution data and improved robustness to adversarial attacks. Additionally, we observe that the predictions of different tasks of this multi-task network are more consistent with each other.