Decision-Making Under Uncertainty for Multi-stage Pipelines: Simulation Studies to Benchmark Screening Strategies

TL;DR

This paper develops models and algorithms to simulate and analyze multi-stage screening pipelines, providing insights into how problem parameters and stage-wise covariance influence screening performance and decision-making under uncertainty.

Contribution

It introduces a general framework for characterizing and benchmarking screening pipeline performance across various problem settings.

Findings

Stage-wise covariance significantly affects screening effectiveness.

Screening performance can sometimes be worse than random selection.

Simulation studies quantify how problem parameters impact pipeline outcomes.

Abstract

Multi-stage screening pipelines are ubiquitous throughout experimental and computational science. Much of the effort in developing screening pipelines focuses on improving generative methods or surrogate models in an attempt to make each screening step effective for a specific application. Little focus has been placed on characterizing generic screening pipeline performance with respect to the problem or problem parameters. Here, we develop models and algorithms to codify and simulate features and properties of the screening procedure in general. We outline and model common problem settings and potential opportunities to perform decision-making under uncertainty to optimize the execution of screening pipelines. We then illustrate the models and algorithms through several simulation studies. We finally show how such studies can provide a quantification of the screening pipeline performance with respect to problem parameters, specifically identifying the significance of stage-wise covariance structure. We show how such structure can lead to qualitatively different screening behaviors, and how screening can even perform worse than random in some cases.