Non-Iterative Knowledge Fusion in Deep Convolutional Neural Networks

TL;DR

This paper introduces two non-iterative methods for fusing knowledge from separate neural networks by combining or modifying weights, enabling efficient transfer without retraining, and demonstrating improved classification performance.

Contribution

The paper presents novel non-iterative techniques for knowledge fusion in neural networks by weight summation and modification, avoiding retraining.

Findings

Fused networks perform significantly better than chance.

Methods work for both shallow and deep neural networks.

Knowledge transfer is achieved without additional training sessions.

Abstract

Incorporation of a new knowledge into neural networks with simultaneous preservation of the previous one is known to be a nontrivial problem. This problem becomes even more complex when new knowledge is contained not in new training examples, but inside the parameters (connection weights) of another neural network. Here we propose and test two methods allowing combining the knowledge contained in separate networks. One method is based on a simple operation of summation of weights of constituent neural networks. Another method assumes incorporation of a new knowledge by modification of weights nonessential for the preservation of already stored information. We show that with these methods the knowledge from one network can be transferred into another one non-iteratively without requiring training sessions. The fused network operates efficiently, performing classification far better than a chance level. The efficiency of the methods is quantified on several publicly available data sets in classification tasks both for shallow and deep neural networks.