AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios

TL;DR

AttNS introduces an attention-inspired approach to improve the generalization and robustness of numerical solvers for differential equations, especially under limited data conditions, by integrating attention mechanisms into the solver design.

Contribution

This paper presents AttNS, a novel attention-inspired numerical solver that enhances generalization and robustness in solving differential equations with limited data, inspired by ResNet's attention modules.

Findings

AttNS improves solver robustness on high-dimensional and chaotic systems.

AttNS requires less data to achieve comparable accuracy.

AttNS prevents numerical explosion issues effectively.

Abstract

We propose the attention-inspired numerical solver (AttNS), a concise method that helps the generalization and robustness issues faced by the AI-Hybrid numerical solver in solving differential equations due to limited data. AttNS is inspired by the effectiveness of attention modules in Residual Neural Networks (ResNet) in enhancing model generalization and robustness for conventional deep learning tasks. Drawing from the dynamical system perspective of ResNet, we seamlessly incorporate attention mechanisms into the design of numerical methods tailored for the characteristics of solving differential equations. Our results on benchmarks, ranging from high-dimensional problems to chaotic systems, showcases AttNS consistently enhancing various numerical solvers without any intricate model crafting. Finally, we analyze AttNS experimentally and theoretically, demonstrating its ability to achieve strong generalization and robustness while ensuring the convergence of the solver. This includes requiring less data compared to other advanced methods to achieve comparable generalization errors and better prevention of numerical explosion issues when solving differential equations.