A Flow Model of Neural Networks

TL;DR

This paper introduces a continuous flow model linking ResNet and transport equations, offering new insights into neural network design and enabling the application of differential equation tools to deep learning.

Contribution

It proposes a novel continuous flow framework for ResNet and plain networks, clarifying their relationship and underlying phenomena in deep neural networks.

Findings

ResNet can be viewed as a refinement of a plain net within the flow model

The flow model explains why 2-layer blocks are necessary in ResNets

Deeper networks benefit from the continuous flow perspective

Abstract

Based on a natural connection between ResNet and transport equation or its characteristic equation, we propose a continuous flow model for both ResNet and plain net. Through this continuous model, a ResNet can be explicitly constructed as a refinement of a plain net. The flow model provides an alternative perspective to understand phenomena in deep neural networks, such as why it is necessary and sufficient to use 2-layer blocks in ResNets, why deeper is better, and why ResNets are even deeper, and so on. It also opens a gate to bring in more tools from the huge area of differential equations.