Cross-stitch Networks for Multi-task Learning

TL;DR

This paper introduces cross-stitch units for multi-task learning in ConvNets, enabling the network to learn optimal shared and task-specific features, leading to improved performance across multiple tasks.

Contribution

It proposes a novel cross-stitch unit that allows end-to-end learning of shared representations in multi-task ConvNets, generalizing across tasks and improving accuracy.

Findings

Significant performance improvements over baselines.

Effective sharing of representations across multiple tasks.

Better generalization with fewer training examples.

Abstract

Multi-task learning in Convolutional Networks has displayed remarkable success in the field of recognition. This success can be largely attributed to learning shared representations from multiple supervisory tasks. However, existing multi-task approaches rely on enumerating multiple network architectures specific to the tasks at hand, that do not generalize. In this paper, we propose a principled approach to learn shared representations in ConvNets using multi-task learning. Specifically, we propose a new sharing unit: "cross-stitch" unit. These units combine the activations from multiple networks and can be trained end-to-end. A network with cross-stitch units can learn an optimal combination of shared and task-specific representations. Our proposed method generalizes across multiple tasks and shows dramatically improved performance over baseline methods for categories with few training examples.