Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning

TL;DR

This paper introduces a zeroth-order optimization approach to integrate scientific knowledge sources with deep learning models, enabling non-intrusive, data-efficient training for complex scientific applications.

Contribution

It presents a novel zeroth-order, gradient-free training scheme that allows seamless integration of black-box scientific knowledge sources with deep learning models.

Findings

Effective integration of scientific knowledge improves model performance in data-limited scenarios.

The proposed method outperforms purely data-driven models in real-world material science applications.

The approach is non-intrusive and adaptable to various knowledge sources.

Abstract

Using deep learning (DL) to accelerate and/or improve scientific workflows can yield discoveries that are otherwise impossible. Unfortunately, DL models have yielded limited success in complex scientific domains due to large data requirements. In this work, we propose to overcome this issue by integrating the abundance of scientific knowledge sources (SKS) with the DL training process. Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm. In contrast, our proposed approach treats knowledge source as a black-box in turn allowing to integrate virtually any knowledge source. To enable an end-to-end training of SKS-coupled-DL, we propose to use zeroth-order optimization (ZOO) based gradient-free training schemes, which is non-intrusive, i.e., does not require making any changes to the SKS. We evaluate the performance of our ZOO training scheme on two real-world material science applications. We show that proposed scheme is able to effectively integrate scientific knowledge with DL training and is able to outperform purely data-driven model for data-limited scientific applications. We also discuss some limitations of the proposed method and mention potentially worthwhile future directions.