FedMef: Towards Memory-efficient Federated Dynamic Pruning
Hong Huang, Weiming Zhuang, Chen Chen, Lingjuan Lyu
TL;DR
FedMef introduces a memory-efficient federated dynamic pruning framework that reduces memory usage by 28.5% and maintains high accuracy, enabling deep learning on resource-constrained devices.
Contribution
The paper proposes FedMef, a novel framework with budget-aware extrusion and scaled activation pruning to enhance efficiency and performance in federated learning.
Findings
Reduces memory footprint by 28.5% compared to state-of-the-art methods.
Maintains or improves model accuracy with efficient pruning.
Demonstrates effectiveness through extensive experiments.
Abstract
Federated learning (FL) promotes decentralized training while prioritizing data confidentiality. However, its application on resource-constrained devices is challenging due to the high demand for computation and memory resources to train deep learning models. Neural network pruning techniques, such as dynamic pruning, could enhance model efficiency, but directly adopting them in FL still poses substantial challenges, including post-pruning performance degradation, high activation memory usage, etc. To address these challenges, we propose FedMef, a novel and memory-efficient federated dynamic pruning framework. FedMef comprises two key components. First, we introduce the budget-aware extrusion that maintains pruning efficiency while preserving post-pruning performance by salvaging crucial information from parameters marked for pruning within a given budget. Second, we propose scaled…
Peer Reviews
Decision·ICLR 2024 Conference Withdrawn Submission
The paper has the following strengths: 1) The paper addresses important challenges in federated learning - accuracy degradation and high memory usage during on-device training. This makes it highly relevant given the growth of federated learning. 2) The proposed techniques - i) budget-aware extrusion and ii) scaled activation pruning - are interesting and provide effective solutions to the mentioned challenges. 3) The methods are well-motivated through mathematical analysis and derivations. For
The paper can be improved based on the following points. 1) The computational overhead of the proposed methods is not analyzed. What is the additional computational cost of budget-aware extrusion and scaled activation pruning? 2) The convergence behavior compared to baselines is not mentioned in the paper. Does your method converge faster or slower than the baselines? 3) Only computer vision tasks are evaluated. Similarly, have you explored your method on other tasks like NLP (non-computer visi
+ The experiments show that FedMeF outperforms recent works like FedTiny and FedDST on the CIFAR-10, CINIC-10 and TinyImageNet datasets interns of accuracy given comparable memory footprints.
- The authors only focused on comparing the pruned models without showing how much communication cost + computation cost was required to achieve such performance. In other words, the cost of pruning, interns of communication and computation cost isn’t discussed. I assume this could be an important metric to consider when comparing against other pruning methods. - Presentation could be improved. For example, Figure 1 could be improved. - The experiment done by the author is not enough and is don
+ The problem this paper focuses on is important and timely for cross-device FL. + The shining point of the work is the two newly proposed components that reduce the activation memory while maintaining or even achieving better performance than the compared counterparts. + The experiments are quite solid. + The paper is well written. The motivation and problem statement are clear and easy to understand.
- The paper mainly focused on cross-device FL, it would be interesting to discuss whether the proposed method is suitable for cross-silo FL. - Some sensitivity analysis is missing, for instance, what is the impact of local epochs and the number of clients selected in each round?
- The activation pruning in FedMef is able to reduce the memory footprint. - FedMef introduces budget-aware extrusion to compensate for post-pruning performance loss. - Adequent reference to discuss related works - Experiments show performance improvement compared to baseline
1. The presentation should be polished. Although the grammar and fluency of the presentation is good, however: - there are a lot of redundant/overlapped sentences paragraph throughout the introduction, methodology, and experiments, which raises unclear and confusion - Figure 1 needs further polished. It shows confusion and unclear, reader can not catch information from it 2. Limited experiments. The experiments cannot support the arguments the paper proposes. 3. For the federated learning opti
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Agent-Based Network Management
MethodsPruning