MetaGater: Fast Learning of Conditional Channel Gated Networks via Federated Meta-Learning

TL;DR

MetaGater introduces a federated meta-learning framework that jointly trains backbone and gating modules, enabling rapid adaptation of efficient, task-specific channel gated networks across distributed nodes.

Contribution

It proposes a novel federated meta-learning approach for jointly training backbone and gating modules, facilitating quick task-specific adaptation in resource-constrained environments.

Findings

Effective in quickly adapting to new tasks with one-step gradient updates

Outperforms existing methods in efficiency and accuracy

Converges under mild theoretical conditions

Abstract

While deep learning has achieved phenomenal successes in many AI applications, its enormous model size and intensive computation requirements pose a formidable challenge to the deployment in resource-limited nodes. There has recently been an increasing interest in computationally-efficient learning methods, e.g., quantization, pruning and channel gating. However, most existing techniques cannot adapt to different tasks quickly. In this work, we advocate a holistic approach to jointly train the backbone network and the channel gating which enables dynamical selection of a subset of filters for more efficient local computation given the data input. Particularly, we develop a federated meta-learning approach to jointly learn good meta-initializations for both backbone networks and gating modules, by making use of the model similarity across learning tasks on different nodes. In this way, the learnt meta-gating module effectively captures the important filters of a good meta-backbone network, based on which a task-specific conditional channel gated network can be quickly adapted, i.e., through one-step gradient descent, from the meta-initializations in a two-stage procedure using new samples of that task. The convergence of the proposed federated meta-learning algorithm is established under mild conditions. Experimental results corroborate the effectiveness of our method in comparison to related work.