Meta Multi-Task Learning for Sequence Modeling

TL;DR

This paper introduces a meta multi-task learning approach for sequence modeling that uses a shared meta-network to generate task-specific composition functions, improving expressiveness and transferability across tasks.

Contribution

It proposes a novel sharing scheme using a meta-network to generate composition functions for multiple tasks, enhancing flexibility and transferability.

Findings

Improved performance on text classification and sequence tagging tasks.

Shared meta-knowledge can be transferred to new tasks.

Demonstrates the effectiveness of meta multi-task learning in sequence modeling.

Abstract

Semantic composition functions have been playing a pivotal role in neural representation learning of text sequences. In spite of their success, most existing models suffer from the underfitting problem: they use the same shared compositional function on all the positions in the sequence, thereby lacking expressive power due to incapacity to capture the richness of compositionality. Besides, the composition functions of different tasks are independent and learned from scratch. In this paper, we propose a new sharing scheme of composition function across multiple tasks. Specifically, we use a shared meta-network to capture the meta-knowledge of semantic composition and generate the parameters of the task-specific semantic composition models. We conduct extensive experiments on two types of tasks, text classification and sequence tagging, which demonstrate the benefits of our approach. Besides, we show that the shared meta-knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks.