Collaborative Adaptive Curriculum for Progressive Knowledge Distillation

TL;DR

This paper introduces FAPD, a federated learning framework that adaptively transfers knowledge by hierarchically decomposing teacher features, improving convergence and accuracy in resource-constrained, heterogeneous environments.

Contribution

It proposes a novel curriculum-inspired, PCA-based hierarchical knowledge transfer method with adaptive pacing, enhancing federated distillation performance.

Findings

Achieves 3.64% higher accuracy than FedAvg on CIFAR-10.

Demonstrates 2x faster convergence compared to fixed-complexity methods.

Maintains robust performance under extreme data heterogeneity ({ extalpha}=0.1).

Abstract

Recent advances in collaborative knowledge distillation have demonstrated cutting-edge performance for resource-constrained distributed multimedia learning scenarios. However, achieving such competitiveness requires addressing a fundamental mismatch: high-dimensional teacher knowledge complexity versus heterogeneous client learning capacities, which currently prohibits deployment in edge-based visual analytics systems. Drawing inspiration from curriculum learning principles, we introduce Federated Adaptive Progressive Distillation (FAPD), a consensus-driven framework that orchestrates adaptive knowledge transfer. FAPD hierarchically decomposes teacher features via PCA-based structuring, extracting principal components ordered by variance contribution to establish a natural visual knowledge hierarchy. Clients progressively receive knowledge of increasing complexity through dimension-adaptive projection matrices. Meanwhile, the server monitors network-wide learning stability by tracking global accuracy fluctuations across a temporal consensus window, advancing curriculum dimensionality only when collective consensus emerges. Consequently, FAPD provably adapts knowledge transfer pace while achieving superior convergence over fixed-complexity approaches. Extensive experiments on three datasets validate FAPD's effectiveness: it attains 3.64% accuracy improvement over FedAvg on CIFAR-10, demonstrates 2x faster convergence, and maintains robust performance under extreme data heterogeneity ({\alpha}=0.1), outperforming baselines by over 4.5%.