Conditional updates of neural network weights for increased out of training performance

TL;DR

This paper introduces a conditional weight update method for neural networks to improve out-of-distribution performance, demonstrated through climate science applications involving temporal, spatial, and cross-domain extrapolations.

Contribution

The paper presents a novel three-step method for adapting neural network weights to out-of-distribution data using regression-based extrapolation.

Findings

Successful out-of-distribution extrapolations in climate science cases

Improved neural network performance on application data

Method applicable to temporal, spatial, and cross-domain shifts

Abstract

This study proposes a method to enhance neural network performance when training data and application data are not very similar, e.g., out of distribution problems, as well as pattern and regime shifts. The method consists of three main steps: 1) Retrain the neural network towards reasonable subsets of the training data set and note down the resulting weight anomalies. 2) Choose reasonable predictors and derive a regression between the predictors and the weight anomalies. 3) Extrapolate the weights, and thereby the neural network, to the application data. We show and discuss this method in three use cases from the climate sciences, which include successful temporal, spatial and cross-domain extrapolations of neural networks.