Theoretical Investigation of Composite Neural Network

TL;DR

This paper provides a theoretical analysis showing that composite neural networks, built from pre-trained models, generally outperform individual components and improve with additional pre-trained modules, supported by empirical evidence.

Contribution

It offers a theoretical proof that composite neural networks outperform their components and benefit from added pre-trained modules, clarifying their performance advantages.

Findings

Composite neural networks outperform individual pre-trained components.

Adding extra pre-trained modules generally improves performance.

Empirical evaluations support the theoretical results.

Abstract

This work theoretically investigates the performance of a composite neural network. A composite neural network is a rooted directed acyclic graph combining a set of pre-trained and non-instantiated neural network models, where a pre-trained neural network model is well-crafted for a specific task and targeted to approximate a specific function with instantiated weights. The advantages of adopting such a pre-trained model in a composite neural network are two folds. One is to benefit from other's intelligence and diligence, and the other is saving the efforts in data preparation and resources and time in training. However, the overall performance of composite neural network is still not clear. In this work, we prove that a composite neural network, with high probability, performs better than any of its pre-trained components under certain assumptions. In addition, if an extra pre-trained component is added to a composite network, with high probability the overall performance will be improved. In the empirical evaluations, distinctively different applications support the above findings.