Growing Neural Network with Shared Parameter

TL;DR

This paper introduces a method for growing neural networks with shared parameters by adding subnetworks based on input matching, supported by Hoeffding's inequality, enabling efficient transfer learning and improved performance.

Contribution

It presents a novel theoretical framework and practical method for expanding neural networks with shared parameters, enhancing transfer learning and parameter efficiency.

Findings

Improves neural network performance by adding subnetworks.

Enables transfer learning without retraining on new tasks.

Achieves higher parameter efficiency in neural network growth.

Abstract

We propose a general method for growing neural network with shared parameter by matching trained network to new input. By leveraging Hoeffding's inequality, we provide a theoretical base for improving performance by adding subnetwork to existing network. With the theoretical base of adding new subnetwork, we implement a matching method to apply trained subnetwork of existing network to new input. Our method has shown the ability to improve performance with higher parameter efficiency. It can also be applied to trans-task case and realize transfer learning by changing the combination of subnetworks without training on new task.