# Incremental Learning Through Deep Adaptation

## TL;DR

The paper introduces Deep Adaptation Networks (DAN), a method for incremental learning that preserves original performance, reduces parameter growth, and enables multi-domain task solving by constraining new filters as linear combinations of existing ones.

## Contribution

DAN provides an efficient incremental learning approach that maintains performance, significantly reduces parameter increase, and allows multi-domain task switching within a single network.

## Key findings

- DAN requires only about 13% of parameters compared to fine-tuning.
- Coupled with quantization, parameter cost drops to around 3%.
- DAN achieves comparable or better performance with fewer training cycles.

## Abstract

Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called \emph{Deep Adaptation Networks} (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs precisely preserve performance on the original domain, require a fraction (typically 13\%, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3\% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.