Decision-Focused Sequential Experimental Design: A Directional Uncertainty-Guided Approach

TL;DR

This paper introduces a novel directional uncertainty metric for sequential experimental design in predict-then-optimize problems, improving decision quality and efficiency over traditional methods through theoretical guarantees and real-world validation.

Contribution

It proposes a computationally tractable directional uncertainty measure and a sequential design criterion with proven consistency and earlier stopping compared to decision-blind approaches.

Findings

The new design attains earlier stopping times under broad distribution classes.

The approach has strong theoretical guarantees of consistency and convergence.

Real-world experiments demonstrate improved decision performance.

Abstract

We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.