Multi-Domain Learning with Modulation Adapters

TL;DR

This paper introduces Modulation Adapters for multi-domain learning, enabling flexible, multiplicative updates to convolutional filters, which improve performance across diverse image classification tasks.

Contribution

The paper proposes Modulation Adapters that update convolutional filters multiplicatively, offering scalable, task-specific adaptation in multi-domain learning.

Findings

Achieves state-of-the-art accuracy on Visual Decathlon and ImageNet-to-Sketch benchmarks.

Flexible parameter scaling balances model complexity and accuracy.

Outperforms existing methods in multi-domain image classification.

Abstract

Deep convolutional networks are ubiquitous in computer vision, due to their excellent performance across different tasks for various domains. Models are, however, often trained in isolation for each task, failing to exploit relatedness between tasks and domains to learn more compact models that generalise better in low-data regimes. Multi-domain learning aims to handle related tasks, such as image classification across multiple domains, simultaneously. Previous work on this problem explored the use of a pre-trained and fixed domain-agnostic base network, in combination with smaller learnable domain-specific adaptation modules. In this paper, we introduce Modulation Adapters, which update the convolutional filter weights of the model in a multiplicative manner for each task. Parameterising these adaptation weights in a factored manner allows us to scale the number of per-task parameters in a flexible manner, and to strike different parameter-accuracy trade-offs. We evaluate our approach on the Visual Decathlon challenge, composed of ten image classification tasks across different domains, and on the ImageNet-to-Sketch benchmark, which consists of six image classification tasks. Our approach yields excellent results, with accuracies that are comparable to or better than those of existing state-of-the-art approaches.