Federated Selective Aggregation for Knowledge Amalgamation

TL;DR

This paper introduces Federated Selective Aggregation (FedSA), a method for training a student model using decentralized teacher models with different tasks, addressing privacy issues and enabling knowledge transfer without model sharing.

Contribution

The paper proposes a novel saliency-based strategy for selective teacher aggregation in federated learning, facilitating knowledge amalgamation from diverse, private models.

Findings

FedSA effectively combines knowledge from decentralized models.

Achieves performance comparable to centralized training baselines.

Works well in both single-task and multi-task scenarios.

Abstract

In this paper, we explore a new knowledge-amalgamation problem, termed Federated Selective Aggregation (FedSA). The goal of FedSA is to train a student model for a new task with the help of several decentralized teachers, whose pre-training tasks and data are different and agnostic. Our motivation for investigating such a problem setup stems from a recent dilemma of model sharing. Many researchers or institutes have spent enormous resources on training large and competent networks. Due to the privacy, security, or intellectual property issues, they are, however, not able to share their own pre-trained models, even if they wish to contribute to the community. The proposed FedSA offers a solution to this dilemma and makes it one step further since, again, the learned student may specialize in a new task different from all of the teachers. To this end, we proposed a dedicated strategy for handling FedSA. Specifically, our student-training process is driven by a novel saliency-based approach that adaptively selects teachers as the participants and integrates their representative capabilities into the student. To evaluate the effectiveness of FedSA, we conduct experiments on both single-task and multi-task settings. Experimental results demonstrate that FedSA effectively amalgamates knowledge from decentralized models and achieves competitive performance to centralized baselines.