Flexible and Adaptable Summarization via Expertise Separation

TL;DR

This paper introduces MoeSumm, a parameter-efficient Mixture-of-Expert architecture that separates general and domain-specific summarization abilities, enhancing flexibility and adaptability across multiple tasks and domains.

Contribution

The paper proposes MoeSumm, a novel mixture-of-experts model with a max-margin loss to explicitly separate general and domain-specific summarization skills, improving multi-domain performance.

Findings

Outperforms recent baselines and LLMs on 11 datasets

Demonstrates effective separation of general and domain-specific abilities

Maintains parameter efficiency while enhancing flexibility and adaptability

Abstract

A proficient summarization model should exhibit both flexibility -- the capacity to handle a range of in-domain summarization tasks, and adaptability -- the competence to acquire new knowledge and adjust to unseen out-of-domain tasks. Unlike large language models (LLMs) that achieve this through parameter scaling, we propose a more parameter-efficient approach in this study. Our motivation rests on the principle that the general summarization ability to capture salient information can be shared across different tasks, while the domain-specific summarization abilities need to be distinct and tailored. Concretely, we propose MoeSumm, a Mixture-of-Expert Summarization architecture, which utilizes a main expert for gaining the general summarization capability and deputy experts that selectively collaborate to meet specific summarization task requirements. We further propose a max-margin loss to stimulate the separation of these abilities. Our model's distinct separation of general and domain-specific summarization abilities grants it with notable flexibility and adaptability, all while maintaining parameter efficiency. MoeSumm achieves flexibility by managing summarization across multiple domains with a single model, utilizing a shared main expert and selected deputy experts. It exhibits adaptability by tailoring deputy experts to cater to out-of-domain few-shot and zero-shot scenarios. Experimental results on 11 datasets show the superiority of our model compared with recent baselines and LLMs. We also provide statistical and visual evidence of the distinct separation of the two abilities in MoeSumm (https://github.com/iriscxy/MoE_Summ).