Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equations

TL;DR

This paper introduces the LySep model, a novel deep learning framework with auxiliary variables that enhances the solution of partial differential equations by improving optimization stability and convergence.

Contribution

The paper proposes a new layer separation approach with auxiliary variables and alternating direction algorithms to improve deep learning optimization for PDEs.

Findings

LySep reduces loss and solution error in high-dimensional PDEs.

Theoretical analysis confirms consistency between LySep and original models.

Numerical results validate the effectiveness of LySep in PDE solving.

Abstract

In this paper, we propose a new optimization framework, the layer separation (LySep) model, to improve the deep learning-based methods in solving partial differential equations. Due to the highly non-convex nature of the loss function in deep learning, existing optimization algorithms often converge to suboptimal local minima or suffer from gradient explosion or vanishing, resulting in poor performance. To address these issues, we introduce auxiliary variables to separate the layers of deep neural networks. Specifically, the output and its derivatives of each layer are represented by auxiliary variables, effectively decomposing the deep architecture into a series of shallow architectures. New loss functions with auxiliary variables are established, in which only variables from two neighboring layers are coupled. Corresponding algorithms based on alternating directions are developed, where many variables can be updated optimally in closed forms. Moreover, we provide theoretical analyses demonstrating the consistency between the LySep model and the original deep model. High-dimensional numerical results validate our theory and demonstrate the advantages of LySep in minimizing loss and reducing solution error.