Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference

TL;DR

This paper introduces a reparameterization technique for convolutions that enables incremental multi-task learning without task interference, achieving state-of-the-art results on benchmark datasets.

Contribution

The authors propose a novel convolution reparameterization that separates shared and task-specific components, allowing incremental learning without performance degradation.

Findings

Achieves state-of-the-art on PASCAL-Context and NYUD benchmarks.

Enables incremental learning without forgetting previous tasks.

Reduces task interference in multi-task networks.

Abstract

Multi-task networks are commonly utilized to alleviate the need for a large number of highly specialized single-task networks. However, two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning). Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference). In this paper, we show that both can be achieved simply by reparameterizing the convolutions of standard neural network architectures into a non-trainable shared part (filter bank) and task-specific parts (modulators), where each modulator has a fraction of the filter bank parameters. Thus, our reparameterization enables the model to learn new tasks without adversely affecting the performance of existing ones. The results of our ablation study attest the efficacy of the proposed reparameterization. Moreover, our method achieves state-of-the-art on two challenging multi-task learning benchmarks, PASCAL-Context and NYUD, and also demonstrates superior incremental learning capability as compared to its close competitors.