Robust Information Selection for Hypothesis Testing with Misclassification Penalties

TL;DR

This paper introduces a robust information selection framework for Bayesian hypothesis testing that accounts for misclassification penalties, proposing algorithms with performance guarantees and demonstrating their effectiveness empirically.

Contribution

It presents a novel misclassification penalty framework and efficient greedy algorithms with guarantees for selecting information sources under uncertainty.

Findings

Algorithms outperform baseline methods in experiments.

The surrogate metric effectively guides source selection.

Proposed methods handle source failures robustly.

Abstract

We study the problem of robust information selection for a Bayesian hypothesis testing / classification task, where the goal is to identify the true state of the world from a finite set of hypotheses based on observations from the selected information sources. We introduce a novel misclassification penalty framework, which enables non-uniform treatment of different misclassification events. Extending the classical subset selection framework, we study the problem of selecting a subset of sources that minimize the maximum penalty of misclassification under a limited budget, despite deletions or failures of a subset of the selected sources. We characterize the curvature properties of the objective function and propose an efficient greedy algorithm with performance guarantees. Next, we highlight certain limitations of optimizing for the maximum penalty metric and propose a submodular surrogate metric to guide the selection of the information set. We propose a greedy algorithm with near-optimality guarantees for optimizing the surrogate metric. Finally, we empirically demonstrate the performance of our proposed algorithms in several instances of the information set selection problem.