Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science

TL;DR

This paper introduces SIB-CL, a contrastive learning framework that leverages auxiliary data, invariances, and surrogate information to significantly reduce the labeled data required for scientific deep learning tasks.

Contribution

SIB-CL is a novel framework that effectively incorporates inexpensive auxiliary information sources to address data scarcity in scientific deep learning applications.

Findings

Achieves orders of magnitude reduction in labeled data needed

Demonstrates effectiveness on photonic crystal and quantum physics problems

Generalizes across different scientific domains

Abstract

Deep learning techniques have been increasingly applied to the natural sciences, e.g., for property prediction and optimization or material discovery. A fundamental ingredient of such approaches is the vast quantity of labelled data needed to train the model; this poses severe challenges in data-scarce settings where obtaining labels requires substantial computational or labor resources. Here, we introduce surrogate- and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three ``inexpensive'' and easily obtainable auxiliary information sources to overcome data scarcity. Specifically, these are: 1)~abundant unlabeled data, 2)~prior knowledge of symmetries or invariances and 3)~surrogate data obtained at near-zero cost. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-of-states of 2D photonic crystals and solving the 3D time-independent Schrodinger equation. SIB-CL consistently results in orders of magnitude reduction in the number of labels needed to achieve the same network accuracies.