# Network with Sub-Networks

## TL;DR

This paper proposes a neural network architecture with detachable sub-networks, allowing parts of the model to be separated during inference without sacrificing accuracy.

## Contribution

It introduces a training method for a base neural network with detachable sub-networks, maintaining accuracy while enabling layer detachment during inference.

## Key findings

- Base model achieves comparable accuracy to standard models.
- Detachable sub-networks can be separated during inference.
- Training involves shared gradients for base and sub-networks.

## Abstract

We introduce network with sub-networks, a neural network which its weight layers could be detached into sub-neural networks during inference. To develop weights and biases which could be inserted in both base and sub-neural networks, firstly, the parameters are copied from sub-model to base-model. Each model is forward-propagated separately. Gradients from a pair of networks are averaged and, used to update both networks. Our base model achieves the test-accuracy which is comparable to the regularly trained models, while the model maintains the ability to detach weight layers.

---
Source: https://tomesphere.com/paper/1908.00763