A Proper Scoring Rule for Virtual Staining

TL;DR

This paper introduces an information gain scoring rule for evaluating virtual staining models in high-throughput screening, enabling direct assessment of predicted biological feature distributions and revealing performance differences missed by traditional metrics.

Contribution

The paper proposes a novel, theoretically sound evaluation framework using information gain for virtual staining models, improving upon existing methods by directly assessing posterior predictions.

Findings

Information gain effectively evaluates virtual staining models.

IG reveals performance differences not captured by other metrics.

Diffusion- and GAN-based models are compared using IG on extensive datasets.

Abstract

Generative virtual staining (VS) models for high-throughput screening (HTS) can provide an estimated posterior distribution of possible biological feature values for each input and cell. However, when evaluating a VS model, the true posterior is unavailable. Existing evaluation protocols only check the accuracy of the marginal distribution over the dataset rather than the predicted posteriors. We introduce information gain (IG) as a cell-wise evaluation framework that enables direct assessment of predicted posteriors. IG is a strictly proper scoring rule and comes with a sound theoretical motivation allowing for interpretability, and for comparing results across models and features. We evaluate diffusion- and GAN-based models on an extensive HTS dataset using IG and other metrics and show that IG can reveal substantial performance differences other metrics cannot.