On the Utility of Virtual Staining for Downstream Applications as it relates to Task Network Capacity

TL;DR

This paper evaluates how virtual staining impacts downstream biomedical tasks like segmentation and classification, emphasizing that the effectiveness depends on the capacity of the neural networks used for these tasks.

Contribution

It systematically investigates the relationship between virtual staining utility and task network capacity, highlighting when virtual staining improves or degrades performance.

Findings

Virtual staining utility depends on task network capacity.

High-capacity networks may not benefit from virtual staining.

Considering network capacity is crucial for virtual staining applications.

Abstract

Virtual staining, or in-silico-labeling, has been proposed to computationally generate synthetic fluorescence images from label-free images by use of deep learning-based image-to-image translation networks. In most reported studies, virtually stained images have been assessed only using traditional image quality measures such as structural similarity or signal-to-noise ratio. However, in biomedical imaging, images are typically acquired to facilitate an image-based inference, which we refer to as a downstream biological or clinical task. This study systematically investigates the utility of virtual staining for facilitating clinically relevant downstream tasks (like segmentation or classification) with consideration of the capacity of the deep neural networks employed to perform the tasks. Comprehensive empirical evaluations were conducted using biological datasets, assessing task performance by use of label-free, virtually stained, and ground truth fluorescence images. The results demonstrated that the utility of virtual staining is largely dependent on the ability of the segmentation or classification task network to extract meaningful task-relevant information, which is related to the concept of network capacity. Examples are provided in which virtual staining does not improve, or even degrades, segmentation or classification performance when the capacity of the associated task network is sufficiently large. The results demonstrate that task network capacity should be considered when deciding whether to perform virtual staining.