Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks

TL;DR

Side-tuning introduces a lightweight additive approach for adapting pre-trained neural networks, outperforming traditional methods like fine-tuning by reducing overfitting and catastrophic forgetting across diverse tasks.

Contribution

The paper proposes side-tuning, a simple additive method for network adaptation that is more effective and robust than existing approaches like fine-tuning and feature extraction.

Findings

Performs well across multiple tasks including vision, NLP, and reinforcement learning.

Reduces overfitting and catastrophic forgetting compared to traditional methods.

Demonstrates consistent improvements in diverse adaptation scenarios.

Abstract

When training a neural network for a desired task, one may prefer to adapt a pre-trained network rather than starting from randomly initialized weights. Adaptation can be useful in cases when training data is scarce, when a single learner needs to perform multiple tasks, or when one wishes to encode priors in the network. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper, we propose a straightforward alternative: side-tuning. Side-tuning adapts a pre-trained network by training a lightweight "side" network that is fused with the (unchanged) pre-trained network via summation. This simple method works as well as or better than existing solutions and it resolves some of the basic issues with fine-tuning, fixed features, and other common approaches. In particular, side-tuning is less prone to overfitting, is asymptotically consistent, and does not suffer from catastrophic forgetting in incremental learning. We demonstrate the performance of side-tuning under a diverse set of scenarios, including incremental learning (iCIFAR, iTaskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.