Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation

TL;DR

Polynomial-augmented neural networks (PANNs) combine deep neural networks with polynomial approximants, using orthogonality constraints and basis pruning to improve function and PDE approximation, especially for limited smoothness functions.

Contribution

The paper introduces PANNs with orthogonality constraints, basis pruning, and polynomial preconditioning, enhancing approximation accuracy and stability over existing methods.

Findings

PANNs outperform DNNs in function approximation and PDE solutions.

Orthogonality constraints improve training stability and accuracy.

PANNs excel in approximating functions with limited smoothness.

Abstract

We present polynomial-augmented neural networks (PANNs), a novel machine learning architecture that combines deep neural networks (DNNs) with a polynomial approximant. PANNs combine the strengths of DNNs (flexibility and efficiency in higher-dimensional approximation) with those of polynomial approximation (rapid convergence rates for smooth functions). To aid in both stable training and enhanced accuracy over a variety of problems, we present (1) a family of orthogonality constraints that impose mutual orthogonality between the polynomial and the DNN within a PANN; (2) a simple basis pruning approach to combat the curse of dimensionality introduced by the polynomial component; and (3) an adaptation of a polynomial preconditioning strategy to both DNNs and polynomials. We test the resulting architecture for its polynomial reproduction properties, ability to approximate both smooth functions and functions of limited smoothness, and as a method for the solution of partial differential equations (PDEs). Through these experiments, we demonstrate that PANNs offer superior approximation properties to DNNs for both regression and the numerical solution of PDEs, while also offering enhanced accuracy over both polynomial and DNN-based regression (each) when regressing functions with limited smoothness.