Network Clustering for Multi-task Learning

TL;DR

This paper introduces a novel cluster layer for multi-task learning that groups related tasks to improve the transition from general to specific representations, enhancing model efficiency.

Contribution

The paper proposes a new cluster layer mechanism that dynamically groups tasks during training to better facilitate the transition from general to task-specific features in MTL.

Findings

Cluster layer improves MTL performance on document classification

Model efficiently transitions from general to specific representations

Experimental results validate the effectiveness of the clustering approach

Abstract

The Multi-Task Learning (MTL) technique has been widely studied by word-wide researchers. The majority of current MTL studies adopt the hard parameter sharing structure, where hard layers tend to learn general representations over all tasks and specific layers are prone to learn specific representations for each task. Since the specific layers directly follow the hard layers, the MTL model needs to estimate this direct change (from general to specific) as well. To alleviate this problem, we introduce the novel cluster layer, which groups tasks into clusters during training procedures. In a cluster layer, the tasks in the same cluster are further required to share the same network. By this way, the cluster layer produces the general presentation for the same cluster, while produces relatively specific presentations for different clusters. As transitions the cluster layers are used between the hard layers and the specific layers. The MTL model thus learns general representations to specific representations gradually. We evaluate our model with MTL document classification and the results demonstrate the cluster layer is quite efficient in MTL.