SkillNet-NLG: General-Purpose Natural Language Generation with a Sparsely Activated Approach

TL;DR

SkillNet-NLG introduces a sparsely activated model for natural language generation that selectively engages relevant skills, outperforming previous methods on multiple Chinese NLG tasks and efficiently adapting to new tasks.

Contribution

It proposes a novel sparsely activated approach for NLG that enables effective skill-specific learning and adaptation within a single model.

Findings

Outperforms previous best methods on four of five Chinese NLG tasks.

Better than two multi-task baselines, comparable to task-specific models.

Excels in adapting to new tasks with improved performance.

Abstract

We present SkillNet-NLG, a sparsely activated approach that handles many natural language generation tasks with one model. Different from traditional dense models that always activate all the parameters, SkillNet-NLG selectively activates relevant parts of the parameters to accomplish a task, where the relevance is controlled by a set of predefined skills. The strength of such model design is that it provides an opportunity to precisely adapt relevant skills to learn new tasks effectively. We evaluate on Chinese natural language generation tasks. Results show that, with only one model file, SkillNet-NLG outperforms previous best performance methods on four of five tasks. SkillNet-NLG performs better than two multi-task learning baselines (a dense model and a Mixture-of-Expert model) and achieves comparable performance to task-specific models. Lastly, SkillNet-NLG surpasses baseline systems when being adapted to new tasks.