Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning

TL;DR

This paper introduces TAAN, a deep learning model that automatically learns task-specific architectures for multi-task learning, enhancing flexibility and performance through adaptive activations and regularization.

Contribution

The paper presents a novel TAAN model that derives task-specific activation functions from data, enabling flexible architecture sharing in multi-task learning.

Findings

TAAN outperforms existing methods in multi-task learning tasks.

Functional regularization improves TAAN's generalization.

Comprehensive experiments validate the effectiveness of the proposed approach.

Abstract

Multi-task learning (MTL) is a common paradigm that seeks to improve the generalization performance of task learning by training related tasks simultaneously. However, it is still a challenging problem to search the flexible and accurate architecture that can be shared among multiple tasks. In this paper, we propose a novel deep learning model called Task Adaptive Activation Network (TAAN) that can automatically learn the optimal network architecture for MTL. The main principle of TAAN is to derive flexible activation functions for different tasks from the data with other parameters of the network fully shared. We further propose two functional regularization methods that improve the MTL performance of TAAN. The improved performance of both TAAN and the regularization methods is demonstrated by comprehensive experiments.