Design of Experiments with Imputable Feature Data: An Entropy-Based Approach

TL;DR

This paper introduces a novel entropy-based framework that jointly optimizes experiment design and missing data imputation, significantly improving efficiency over traditional sequential methods especially in complex, data-scarce scenarios.

Contribution

It proposes a unified maximum-entropy approach that simultaneously addresses experiment selection and data imputation, overcoming limitations of sequential methods and NP-hard complexity.

Findings

Achieves over 60% improvement in cost value compared to benchmarks.

Effectively handles missing data in experiment design scenarios.

Flexible framework adaptable to various application constraints.

Abstract

Tactical selection of experiments to estimate an underlying model is an innate task across various fields. Since each experiment has costs associated with it, selecting statistically significant experiments becomes necessary. Classic linear experimental design deals with experiment selection so as to minimize (functions of) variance in estimation of regression parameter. Typically, standard algorithms for solving this problem assume that data associated with each experiment is fully known. This isn't often true since missing data is a common problem. For instance, remote sensors often miss data due to poor connection. Hence experiment selection under such scenarios is a widespread but challenging task. Though decoupling the tasks and using standard data imputation methods like matrix completion followed by experiment selection might seem a way forward, they perform sub-optimally since the tasks are naturally interdependent. Standard design of experiments is an NP hard problem, and the additional objective of imputing for missing data amplifies the computational complexity. In this paper, we propose a maximum-entropy-principle based framework that simultaneously addresses the problem of design of experiments as well as the imputation of missing data. Our algorithm exploits homotopy from a suitably chosen convex function to the non-convex cost function; hence avoiding poor local minima. Further, our proposed framework is flexible to incorporate additional application specific constraints. Simulations on various datasets show improvement in the cost value by over 60% in comparison to benchmark algorithms applied sequentially to the imputation and experiment selection problems.