Efficient Multi-Domain Network Learning by Covariance Normalization

TL;DR

This paper introduces CovNorm, a data-driven covariance normalization method for efficient multi-domain deep network learning, significantly reducing parameters while maintaining performance comparable to full fine-tuning.

Contribution

The paper proposes CovNorm, a novel covariance normalization technique that simplifies multi-domain learning by reducing parameters and is effective both sequentially and simultaneously.

Findings

CovNorm achieves comparable performance to full fine-tuning.

It uses only 0.13% of parameters per domain.

The method is theoretically justified and experimentally validated.

Abstract

The problem of multi-domain learning of deep networks is considered. An adaptive layer is induced per target domain and a novel procedure, denoted covariance normalization (CovNorm), proposed to reduce its parameters. CovNorm is a data driven method of fairly simple implementation, requiring two principal component analyzes (PCA) and fine-tuning of a mini-adaptation layer. Nevertheless, it is shown, both theoretically and experimentally, to have several advantages over previous approaches, such as batch normalization or geometric matrix approximations. Furthermore, CovNorm can be deployed both when target datasets are available sequentially or simultaneously. Experiments show that, in both cases, it has performance comparable to a fully fine-tuned network, using as few as 0.13% of the corresponding parameters per target domain.