Universal Representations: A Unified Look at Multiple Task and Domain Learning

TL;DR

This paper introduces a method for learning universal representations in a single neural network that can handle multiple vision tasks and domains, improving performance and efficiency.

Contribution

It proposes a knowledge distillation approach with adapters to unify multiple task/domain-specific networks into one, achieving state-of-the-art results.

Findings

State-of-the-art performance on NYU-v2 and Cityscapes dense prediction tasks.

Superior results on Visual Decathlon Dataset classification problems.

Effective cross-domain few-shot learning in MetaDataset.

Abstract

We propose a unified look at jointly learning multiple vision tasks and visual domains through universal representations, a single deep neural network. Learning multiple problems simultaneously involves minimizing a weighted sum of multiple loss functions with different magnitudes and characteristics and thus results in unbalanced state of one loss dominating the optimization and poor results compared to learning a separate model for each problem. To this end, we propose distilling knowledge of multiple task/domain-specific networks into a single deep neural network after aligning its representations with the task/domain-specific ones through small capacity adapters. We rigorously show that universal representations achieve state-of-the-art performances in learning of multiple dense prediction problems in NYU-v2 and Cityscapes, multiple image classification problems from diverse domains in Visual Decathlon Dataset and cross-domain few-shot learning in MetaDataset. Finally we also conduct multiple analysis through ablation and qualitative studies.