Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions

TL;DR

This paper provides a comprehensive review of knowledge distillation in collaborative machine learning, focusing on how memory and knowledge are managed and interact across various learning paradigms and heterogeneity challenges.

Contribution

It categorizes and analyzes the roles of memory and knowledge in KD within diverse collaborative learning structures, highlighting challenges and future research directions.

Findings

Categorizes memory and knowledge in KD processes.

Analyzes effects of heterogeneity on KD effectiveness.

Provides insights into future research directions.

Abstract

Collaborative learning has emerged as a key paradigm in large-scale intelligent systems, enabling distributed agents to cooperatively train their models while addressing their privacy concerns. Central to this paradigm is knowledge distillation (KD), a technique that facilitates efficient knowledge transfer among agents. However, the underlying mechanisms by which KD leverages memory and knowledge across agents remain underexplored. This paper aims to bridge this gap by offering a comprehensive review of KD in collaborative learning, with a focus on the roles of memory and knowledge. We define and categorize memory and knowledge within the KD process and explore their interrelationships, providing a clear understanding of how knowledge is extracted, stored, and shared in collaborative settings. We examine various collaborative learning patterns, including distributed, hierarchical, and decentralized structures, and provide insights into how memory and knowledge dynamics shape the effectiveness of KD in collaborative learning. Particularly, we emphasize task heterogeneity in distributed learning pattern covering federated learning (FL), multi-agent domain adaptation (MADA), federated multi-modal learning (FML), federated continual learning (FCL), federated multi-task learning (FMTL), and federated graph knowledge embedding (FKGE). Additionally, we highlight model heterogeneity, data heterogeneity, resource heterogeneity, and privacy concerns of these tasks. Our analysis categorizes existing work based on how they handle memory and knowledge. Finally, we discuss existing challenges and propose future directions for advancing KD techniques in the context of collaborative learning.