Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization

TL;DR

This paper introduces AdaFedSelecKD, a federated learning approach for meeting summarization that reduces communication costs and handles non-IID data across clients, achieving centralized-level performance.

Contribution

The paper proposes an adapter-based federated summarization model with selective knowledge distillation to efficiently handle non-IID meeting data in a privacy-preserving manner.

Findings

Achieves comparable performance to centralized models on QMSum benchmark.

Reduces communication costs via adapter-based model design.

Demonstrates robustness and generalizability across domains.

Abstract

Meeting summarization has emerged as a promising technique for providing users with condensed summaries. However, existing work has focused on training models on centralized data, neglecting real-world scenarios where meeting data are infeasible to collect centrally, due to their sensitive nature. This gap motivates us to explore federated learning for meeting summarization. Two critical challenges impede progress. First, state-of-the-art summarizers are based on parameter-heavy pre-trained models. Exchanging such a model's parameters across clients imposes large bandwidth costs. Second, as real-world meeting data belong to various domains and are distributed across clients, they are instances of non-identically and independently distributed (non-IID). IID assumptions do not hold, which changes which forms of learning algorithms best apply. To address this, we propose Adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models. Specifically, we develop an adapter-based summarization model where two adapters cooperatively facilitate learning using fewer parameters to reduce communication costs. Then, we devise a selective knowledge distillation strategy, assisting clients in robustly handling domain-focused modelling on their own data, while leveraging global parameters based on non-IID data. Extensive experiments on the QMSum benchmark demonstrate AdaFedSelecKD can achieve comparable performance with powerful centralized training methods, and shows its generalizability and robustness.