A Kernel Perspective on Distillation-based Collaborative Learning

TL;DR

This paper provides a theoretical analysis of distillation-based collaborative learning from a kernel perspective, proving near-optimality in heterogeneous environments and proposing a neural network-based algorithm validated through simulations.

Contribution

It offers the first theoretical proof of near-minimax optimality for a nonparametric collaborative learning algorithm without data sharing, and introduces a practical neural network method inspired by this theory.

Findings

Theoretical proof of near-minimax optimality in heterogeneous settings.

Proposed neural network algorithm bridges theory and practice.

Simulation results verify the effectiveness of the proposed method.

Abstract

Over the past decade, there is a growing interest in collaborative learning that can enhance AI models of multiple parties. However, it is still challenging to enhance performance them without sharing private data and models from individual parties. One recent promising approach is to develop distillation-based algorithms that exploit unlabeled public data but the results are still unsatisfactory in both theory and practice. To tackle this problem, we rigorously analyze a representative distillation-based algorithm in the view of kernel regression. This work provides the first theoretical results to prove the (nearly) minimax optimality of the nonparametric collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture. Our algorithm successfully bridges the gap between our theoretical assumptions and practical settings with neural networks through feature kernel matching. We simulate various regression tasks to verify our theory and demonstrate the practical feasibility of our proposed algorithm.