AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning

TL;DR

AutoMTL is a programming framework that automates the creation of efficient multi-task learning models for vision tasks, optimizing for both accuracy and memory usage across arbitrary CNN backbones.

Contribution

It introduces the first automated framework for multi-task learning model development that adapts to any CNN backbone without manual re-implementation.

Findings

AutoMTL outperforms state-of-the-art methods on benchmark datasets.

AutoMTL generalizes well across different CNN architectures.

The framework produces models with high accuracy and low memory footprint.

Abstract

Multi-task learning (MTL) jointly learns a set of tasks by sharing parameters among tasks. It is a promising approach for reducing storage costs while improving task accuracy for many computer vision tasks. The effective adoption of MTL faces two main challenges. The first challenge is to determine what parameters to share across tasks to optimize for both memory efficiency and task accuracy. The second challenge is to automatically apply MTL algorithms to an arbitrary CNN backbone without requiring time-consuming manual re-implementation and significant domain expertise. This paper addresses the challenges by developing the first programming framework AutoMTL that automates efficient MTL model development for vision tasks. AutoMTL takes as inputs an arbitrary backbone convolutional neural network (CNN) and a set of tasks to learn, and automatically produces a multi-task model that achieves high accuracy and small memory footprint simultaneously. Experiments on three popular MTL benchmarks (CityScapes, NYUv2, Tiny-Taskonomy) demonstrate the effectiveness of AutoMTL over state-of-the-art approaches as well as the generalizability of AutoMTL across CNNs. AutoMTL is open-sourced and available at https://github.com/zhanglijun95/AutoMTL.