Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation

TL;DR

This paper introduces MemIML, a meta-learning approach that improves low-resource NLP tasks by using a memory imitation module to prevent overfitting and enhance reliance on support sets during adaptation.

Contribution

The paper proposes a novel memory imitation meta-learning method with a task-specific memory module and imitation mechanism to address overfitting in low-resource NLP tasks.

Findings

Outperforms baseline methods on text classification tasks.

Enhances model reliance on support sets during adaptation.

Proves effectiveness through theoretical analysis and empirical results.

Abstract

Building models of natural language processing (NLP) is challenging in low-resource scenarios where only limited data are available. Optimization-based meta-learning algorithms achieve promising results in low-resource scenarios by adapting a well-generalized model initialization to handle new tasks. Nonetheless, these approaches suffer from the memorization overfitting issue, where the model tends to memorize the meta-training tasks while ignoring support sets when adapting to new tasks. To address this issue, we propose a memory imitation meta-learning (MemIML) method that enhances the model's reliance on support sets for task adaptation. Specifically, we introduce a task-specific memory module to store support set information and construct an imitation module to force query sets to imitate the behaviors of some representative support-set samples stored in the memory. A theoretical analysis is provided to prove the effectiveness of our method, and empirical results also demonstrate that our method outperforms competitive baselines on both text classification and generation tasks.