TaskDrop: A Competitive Baseline for Continual Learning of Sentiment Classification

TL;DR

This paper introduces TaskDrop, a simple yet effective method for continual sentiment classification that uses random task-wise capacity masks, achieving competitive results across multiple datasets and demonstrating robustness in long-term learning.

Contribution

The paper proposes TaskDrop, a novel random capacity allocation method for continual sentiment learning, which outperforms or matches existing approaches without complex training objectives.

Findings

TaskDrop achieves competitive performance across three datasets.

It maintains robustness in long-term continual learning.

Random task-wise masking effectively reduces forgetting.

Abstract

In this paper, we study the multi-task sentiment classification problem in the continual learning setting, i.e., a model is sequentially trained to classifier the sentiment of reviews of products in a particular category. The use of common sentiment words in reviews of different product categories leads to large cross-task similarity, which differentiates it from continual learning in other domains. This knowledge sharing nature renders forgetting reduction focused approaches less effective for the problem under consideration. Unlike existing approaches, where task-specific masks are learned with specifically presumed training objectives, we propose an approach called Task-aware Dropout (TaskDrop) to generate masks in a random way. While the standard dropout generates and applies random masks for each training instance per epoch for effective regularization, TaskDrop applies random masking for task-wise capacity allocation and reuse. We conducted experimental studies on three multi-task review datasets and made comparison to various baselines and state-of-the-art approaches. Our empirical results show that regardless of simplicity, TaskDrop overall achieved competitive performances for all the three datasets, especially after relative long term learning. This demonstrates that the proposed random capacity allocation mechanism works well for continual sentiment classification.