Introduction to optimization methods for training SciML models

TL;DR

This paper provides a unified overview of optimization methods tailored for training scientific machine learning models, highlighting differences from classical ML and discussing suitable algorithms for physics-informed problems.

Contribution

It offers a comprehensive review of optimization techniques adapted for SciML, emphasizing problem structure and practical strategies for physics-constrained models.

Findings

Standard stochastic methods often limited in SciML due to problem structure

Deterministic and curvature-aware methods show promise for SciML optimization

Tutorial examples illustrate practical application of discussed optimization strategies

Abstract

Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.