Enhancing Physics-Informed Neural Networks Through Feature Engineering
Shaghayegh Fazliani, Zachary Frangella, Madeleine Udell
TL;DR
This paper introduces SAFE-NET, a simple and efficient feature engineering approach for PINNs that achieves faster convergence, lower errors, and fewer parameters compared to complex architectures, challenging the need for deep networks in scientific computing.
Contribution
SAFE-NET demonstrates that a single-layer network with Fourier features and optimized training can outperform deeper models in PINNs, reducing complexity and training time.
Findings
SAFE-NET converges faster than deeper networks.
SAFE-NET uses 65% fewer parameters on average.
SAFE-NET epochs are 95% faster than competing methods.
Abstract
Physics-Informed Neural Networks (PINNs) seek to solve partial differential equations (PDEs) with deep learning. Mainstream approaches that deploy fully-connected multi-layer deep learning architectures require prolonged training to achieve even moderate accuracy, while recent work on feature engineering allows higher accuracy and faster convergence. This paper introduces SAFE-NET, a Single-layered Adaptive Feature Engineering NETwork that achieves orders-of-magnitude lower errors with far fewer parameters than baseline feature engineering methods. SAFE-NET returns to basic ideas in machine learning, using Fourier features, a simplified single hidden layer network architecture, and an effective optimizer that improves the conditioning of the PINN optimization problem. Numerical results show that SAFE-NET converges faster and typically outperforms deeper networks and more complex…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
The authors introduce SAFE-NET, a model that achieves high efficiency by requiring fewer parameters than existing PINN models. It incorporates feature engineering through Fourier transforms, enabling a compact and powerful model design. By combining well-known optimizers, the model achieves faster training times and improved overall accuracy.
The paper lacks clarity in some explanations, making it challenging to fully understand key details. Specific areas needing improvement are: **1. Data Processing**: The paper does not clearly explain how data is processed and fed into the model. It appears that Fourier transforms are used to focus on specific frequencies within the data, which may imply that SAFE-NET selectively captures frequency information. Could authors explain the exact feature engineering process, including any frequency
S1) The authors provide a parameter efficient (small model) approach for PINNs. Opposed to the larger model sizes in the traditional PINNs, the authors suggest a single-layer MLP network for PINNs S2) A newer hybrid optimizer combining existing optimizers such as Adam and L-BFGS which converges faster and matches or exceeds baseline performance has been used. S3) A major supporting factor and advantage for the experiments done with the new optimiser are the spectral density plots which visuali
W1) There is no mention of the types of scenarios, equations etc. where the method fails (and where it works) such types of equations, regimes, initial or boundary conditions. The method does work on the canonical examples used in the experiments, however how does the feature engineering help with the learning is not clear. Look at the question on the Logarithmic and Polynomial features (Q1) W2) It has been mentioned multiple times that the network is a single layer network, however there is no
The paper's primary strength lies in its comprehensive demonstration that engineered features can significantly enhance PINN performance. While this concept isn't novel and has been explored in previous literature, the authors contribute by providing a systematic evaluation of different feature engineering approaches and their combinations. Their empirical investigation includes comparative analysis of Consecutive Integer Features (CIF), Uniformly-Chosen Integer Features (UIF), Uniformly-Chosen
While the key idea of using feature engineering for improving the performance of PINNs has merit, I have several major concerns about the current presentation and experimentation presented in the manuscript: 1. The experimental comparisons rely heavily on vanilla MLP networks, which are known to be a very weak baseline for PINNs. The absence of comparisons against state-of-the-art methods like PirateNets [1] makes it difficult to assess the true significance of SAFE-NET's contributions. A more
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputational Physics and Python Applications · Neural Networks and Applications