Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

TL;DR

This paper introduces Piggyback, a method that enables a single neural network to adapt to multiple tasks by learning binary masks, avoiding catastrophic forgetting and maintaining performance across diverse tasks with minimal overhead.

Contribution

The paper proposes a novel mask-based approach for multi-task adaptation that preserves existing task performance and is agnostic to task order, unlike prior methods.

Findings

Achieves performance comparable to dedicated fine-tuned networks.

Avoids catastrophic forgetting across multiple tasks.

Requires only 1 bit per parameter per task, with low overhead.

Abstract

This work presents a method for adapting a single, fixed deep neural network to multiple tasks without affecting performance on already learned tasks. By building upon ideas from network quantization and pruning, we learn binary masks that piggyback on an existing network, or are applied to unmodified weights of that network to provide good performance on a new task. These masks are learned in an end-to-end differentiable fashion, and incur a low overhead of 1 bit per network parameter, per task. Even though the underlying network is fixed, the ability to mask individual weights allows for the learning of a large number of filters. We show performance comparable to dedicated fine-tuned networks for a variety of classification tasks, including those with large domain shifts from the initial task (ImageNet), and a variety of network architectures. Unlike prior work, we do not suffer from catastrophic forgetting or competition between tasks, and our performance is agnostic to task ordering. Code available at https://github.com/arunmallya/piggyback.