HeunNet: Extending ResNet using Heun's Methods

TL;DR

HeunNet introduces a predictor-corrector variant of ResNet inspired by Heun's method, achieving higher accuracy with less computational cost by extending the analogy between ResNets and ODE solvers.

Contribution

This paper proposes HeunNet, a novel ResNet variant based on Heun's method, enhancing accuracy and efficiency over existing ResNet models.

Findings

HeunNet outperforms vanilla ResNet in accuracy.

HeunNet requires less training and testing time.

HeunNet demonstrates improved efficiency over other ResNet variants.

Abstract

There is an analogy between the ResNet (Residual Network) architecture for deep neural networks and an Euler solver for an ODE. The transformation performed by each layer resembles an Euler step in solving an ODE. We consider the Heun Method, which involves a single predictor-corrector cycle, and complete the analogy, building a predictor-corrector variant of ResNet, which we call a HeunNet. Just as Heun's method is more accurate than Euler's, experiments show that HeunNet achieves high accuracy with low computational (both training and test) time compared to both vanilla recurrent neural networks and other ResNet variants.