Learning multiple visual domains with residual adapters

TL;DR

This paper introduces a residual adapter-based deep network architecture that enables a single model to effectively learn and adapt to multiple diverse visual domains, improving parameter efficiency and accuracy.

Contribution

It proposes a novel residual adapter module for multi-domain visual learning and introduces the Visual Decathlon Challenge for benchmarking such models.

Findings

High parameter sharing with maintained or improved accuracy

Effective adaptation across ten diverse visual domains

Introduction of a new benchmark for multi-domain visual recognition

Abstract

There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to recognize well uniformly.