Exploring Semantic Attributes from A Foundation Model for Federated Learning of Disjoint Label Spaces

TL;DR

This paper introduces a federated zero-shot learning framework that transfers privacy-preserving mid-level semantic attributes across distributed clients to build a generalizable global model, enhancing scalability and privacy.

Contribution

It formulates a novel federated zero-shot learning paradigm focusing on mid-level semantic knowledge transfer, with semantic augmentation from external sources to improve model discrimination.

Findings

Effective in five zero-shot learning benchmarks

Enriches semantic space with external knowledge

Improves model generalization and privacy preservation

Abstract

Conventional centralised deep learning paradigms are not feasible when data from different sources cannot be shared due to data privacy or transmission limitation. To resolve this problem, federated learning has been introduced to transfer knowledge across multiple sources (clients) with non-shared data while optimising a globally generalised central model (server). Existing federated learning paradigms mostly focus on transferring holistic high-level knowledge (such as class) across models, which are closely related to specific objects of interest so may suffer from inverse attack. In contrast, in this work, we consider transferring mid-level semantic knowledge (such as attribute) which is not sensitive to specific objects of interest and therefore is more privacy-preserving and scalable. To this end, we formulate a new Federated Zero-Shot Learning (FZSL) paradigm to learn mid-level semantic knowledge at multiple local clients with non-shared local data and cumulatively aggregate a globally generalised central model for deployment. To improve model discriminative ability, we propose to explore semantic knowledge augmentation from external knowledge for enriching the mid-level semantic space in FZSL. Extensive experiments on five zeroshot learning benchmark datasets validate the effectiveness of our approach for optimising a generalisable federated learning model with mid-level semantic knowledge transfer.