CoGraM: Context-sensitive granular optimization method with rollback for robust model fusion

TL;DR

CoGraM is a multi-stage, context-aware optimization technique for neural network merging that enhances accuracy and stability in federated learning by using loss-based decisions and rollback mechanisms.

Contribution

It introduces a novel, context-sensitive, iterative optimization method with rollback to improve neural network merging without retraining.

Findings

Significantly improves merged network accuracy.

Enhances stability across different seeds.

Addresses weaknesses of Fisher merging.

Abstract

Merging neural networks without retraining is central to federated and distributed learning. Common methods such as weight averaging or Fisher merging often lose accuracy and are unstable across seeds. CoGraM (Contextual Granular Merging) is a multi-stage, context-sensitive, loss-based, and iterative optimization method across layers, neurons, and weight levels that aligns decisions with loss differences and thresholds and prevents harmful updates through rollback. CoGraM is an optimization method that addresses the weaknesses of methods such as Fisher and can significantly improve the merged network.