Continual Task Allocation in Meta-Policy Network via Sparse Prompting

TL;DR

This paper introduces CoTASP, a method for continual learning in reinforcement learning that uses sparse prompts and sub-networks to adapt to new tasks efficiently while retaining previous knowledge, without replaying past experiences.

Contribution

It proposes a novel sparse prompting approach with over-complete dictionaries for effective task-specific sub-network extraction in meta-policy networks.

Findings

Outperforms existing methods on all seen tasks.

Reduces forgetting effectively.

Generalizes well to unseen tasks.

Abstract

How to train a generalizable meta-policy by continually learning a sequence of tasks? It is a natural human skill yet challenging to achieve by current reinforcement learning: the agent is expected to quickly adapt to new tasks (plasticity) meanwhile retaining the common knowledge from previous tasks (stability). We address it by "Continual Task Allocation via Sparse Prompting (CoTASP)", which learns over-complete dictionaries to produce sparse masks as prompts extracting a sub-network for each task from a meta-policy network. CoTASP trains a policy for each task by optimizing the prompts and the sub-network weights alternatively. The dictionary is then updated to align the optimized prompts with tasks' embedding, thereby capturing tasks' semantic correlations. Hence, relevant tasks share more neurons in the meta-policy network due to similar prompts while cross-task interference causing forgetting is effectively restrained. Given a meta-policy and dictionaries trained on previous tasks, new task adaptation reduces to highly efficient sparse prompting and sub-network finetuning. In experiments, CoTASP achieves a promising plasticity-stability trade-off without storing or replaying any past tasks' experiences. It outperforms existing continual and multi-task RL methods on all seen tasks, forgetting reduction, and generalization to unseen tasks.