Learning Twofold Heterogeneous Multi-Task by Sharing Similar Convolution Kernel Pairs

TL;DR

This paper introduces a novel multi-task learning network that effectively handles twofold heterogeneous scenarios by sharing similar convolution kernels, improving cross-task learning in complex input-output space mismatches.

Contribution

The paper proposes a simple, effective multi-task adaptive learning network that shares similar convolution kernels to address twofold heterogeneity in input and output spaces.

Findings

Outperforms state-of-the-art methods in experiments

Effectively shares knowledge across heterogeneous tasks

Suppresses intra-redundancy in the network

Abstract

Heterogeneous multi-task learning (HMTL) is an important topic in multi-task learning (MTL). Most existing HMTL methods usually solve either scenario where all tasks reside in the same input (feature) space yet unnecessarily the consistent output (label) space or scenario where their input (feature) spaces are heterogeneous while the output (label) space is consistent. However, to the best of our knowledge, there is limited study on twofold heterogeneous MTL (THMTL) scenario where the input and the output spaces are both inconsistent or heterogeneous. In order to handle this complicated scenario, in this paper, we design a simple and effective multi-task adaptive learning (MTAL) network to learn multiple tasks in such THMTL setting. Specifically, we explore and utilize the inherent relationship between tasks for knowledge sharing from similar convolution kernels in individual layers of the MTAL network. Then in order to realize the sharing, we weightedly aggregate any pair of convolutional kernels with their similarity greater than some threshold $\rho$, consequently, our model effectively performs cross-task learning while suppresses the intra-redundancy of the entire network. Finally, we conduct end-to-end training. Our experimental results demonstrate the effectiveness of our method in comparison with the state-of-the-art counterparts.