UniT: Multimodal Multitask Learning with a Unified Transformer

TL;DR

UniT introduces a unified transformer model capable of multitask learning across diverse domains like object detection and natural language understanding, achieving strong performance with fewer parameters by sharing a single model for multiple tasks.

Contribution

The paper presents UniT, a novel unified transformer architecture that jointly learns multiple tasks across different domains with shared parameters, unlike previous models that fine-tune task-specific transformers.

Findings

Achieves strong performance on 7 tasks across 8 datasets.

Uses significantly fewer parameters than task-specific models.

Successfully handles diverse multimodal and multitask learning scenarios.

Abstract

We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning. Based on the transformer encoder-decoder architecture, our UniT model encodes each input modality with an encoder and makes predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. Compared to previous efforts on multi-task learning with transformers, we share the same model parameters across all tasks instead of separately fine-tuning task-specific models and handle a much higher variety of tasks across different domains. In our experiments, we learn 7 tasks jointly over 8 datasets, achieving strong performance on each task with significantly fewer parameters. Our code is available in MMF at https://mmf.sh.