Advances and Open Challenges in Federated Foundation Models

TL;DR

This paper surveys Federated Foundation Models (FedFM), discussing their integration, challenges, methodologies, and future directions, highlighting the importance of privacy, scalability, and trustworthiness in federated AI systems.

Contribution

It provides a comprehensive taxonomy, analyzes key challenges, and offers future research directions for the emerging field of FedFM, integrating federated learning with foundation models.

Findings

Proposes a multi-tiered taxonomy for FedFM approaches

Identifies key challenges like privacy, scalability, and communication efficiency

Highlights potential of quantum computing in FedFM processes

Abstract

The integration of Foundation Models (FMs) with Federated Learning (FL) presents a transformative paradigm in Artificial Intelligence (AI). This integration offers enhanced capabilities, while addressing concerns of privacy, data decentralization and computational efficiency. This paper provides a comprehensive survey of the emerging field of Federated Foundation Models (FedFM), elucidating their synergistic relationship and exploring novel methodologies, challenges, and future directions that the FL research field needs to focus on in order to thrive in the age of FMs. A systematic multi-tiered taxonomy is proposed, categorizing existing FedFM approaches for model training, aggregation, trustworthiness, and incentivization. Key challenges, including how to enable FL to deal with high complexity of computational demands, privacy considerations, contribution evaluation, and communication efficiency, are thoroughly discussed. Moreover, this paper explores the intricate challenges of communication, scalability and security inherent in training/fine-tuning FMs via FL. It highlights the potential of quantum computing to revolutionize the processes of training, inference, optimization and security. This survey also introduces the implementation requirement of FedFM and some practical FedFM applications. It highlights lessons learned with a clear understanding of our findings for FedFM. Finally, this survey not only provides insights into the current state and challenges of FedFM, but also offers a blueprint for future research directions, emphasizing the need for developing trustworthy solutions. It serves as a foundational guide for researchers and practitioners interested in contributing to this interdisciplinary and rapidly advancing field.