HiPreNets: High-Precision Neural Networks through Progressive Training

TL;DR

HiPreNets introduces a progressive training framework for neural networks that sequentially refines models to achieve high-precision results, especially in complex and safety-critical applications.

Contribution

The paper presents a novel progressive training approach that reduces both average and worst-case errors through residual refinements and targeted high-error region training.

Findings

Outperforms standard networks on Feynman dataset benchmarks.

Achieves near machine precision accuracy in regression tasks.

Reduces RMSE and $L^{ty}$ error significantly in high-dimensional power system modeling.

Abstract

Deep neural networks are powerful tools for solving nonlinear problems in science and engineering, but training highly accurate models becomes challenging as problem complexity increases. Non-convex optimization and sensitivity to hyperparameters make consistent performance improvement difficult, and traditional approaches prioritize minimizing mean squared error while overlooking the $L^{\infty}$ norm error that is critical in safety-sensitive applications. To address these challenges, we present HiPreNets, a progressive framework for training high-precision neural networks through sequential residual refinements. Starting from an initial network, each stage trains a refinement network on the normalized residuals of the ensemble so far, systematically reducing both average and worst-case error. A key theme throughout the framework is concentrating training effort on high-error regions of the input domain, which we pursue through complementary techniques including loss function design, adaptive data sampling, localized patching, and boundary-aware training. We validate the framework on benchmark regression problems from the Feynman dataset, where it consistently outperforms standard fully connected networks and reported Kolmogorov-Arnold Networks results, with accuracy approaching machine precision depending on select problems. We further apply the framework to learning the flow map of a 20-dimensional power system ODE, which appears to be the highest dimensional problem studied using this class of multistage methods, achieving substantial reductions in both RMSE and $L^{\infty}$ norm error while enabling a surrogate that predicts system state $238\times$ faster than direct numerical simulation.