Learning Deep Hybrid Models with Sharpness-Aware Minimization

TL;DR

This paper introduces a novel approach to hybrid modeling by applying sharpness-aware minimization to improve model simplicity and robustness, demonstrated through experiments across various models and datasets.

Contribution

It adapts sharpness-aware minimization to hybrid models, enhancing their stability and interpretability regardless of model architecture or domain knowledge.

Findings

SAM-based method improves hybrid model robustness

Effective across different models and datasets

Promotes simpler, more stable solutions

Abstract

Hybrid modeling, the combination of machine learning models and scientific mathematical models, enables flexible and robust data-driven prediction with partial interpretability. However, effectively the scientific models may be ignored in prediction due to the flexibility of the machine learning model, making the idea of hybrid modeling pointless. Typically some regularization is applied to hybrid model learning to avoid such a failure case, but the formulation of the regularizer strongly depends on model architectures and domain knowledge. In this paper, we propose to focus on the flatness of loss minima in learning hybrid models, aiming to make the model as simple as possible. We employ the idea of sharpness-aware minimization and adapt it to the hybrid modeling setting. Numerical experiments show that the SAM-based method works well across different choices of models and datasets.