Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

TL;DR

This paper reveals that many ML-based PDE solving studies for fluid dynamics rely on weak baselines and suffer from reporting biases, leading to overly optimistic claims about their effectiveness.

Contribution

It provides a systematic review highlighting prevalent weak baselines and biases in ML-PDE research, and advocates for cultural and structural reforms to improve scientific integrity.

Findings

79% of studies compare to weak baselines

Widespread outcome and publication biases identified

Overoptimism in reported ML-PDE performance

Abstract

One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.