Optimal Decision Rules for Composite Binary Hypothesis Testing under Neyman-Pearson Framework

TL;DR

This paper develops optimal decision rules for composite binary hypothesis testing under the Neyman-Pearson framework, maximizing a nonlinear detection function while controlling false alarms, with applications to exponential families.

Contribution

It introduces a generalized Bayes rule approach for composite hypotheses and derives optimal threshold rules for various constraints, extending classical methods.

Findings

Optimal single-threshold rules based on weighted likelihood ratios.

Extension to composite null hypotheses with average and worst-case constraints.

Numerical examples illustrating the theoretical optimal rules.

Abstract

The composite binary hypothesis testing problem within the Neyman-Pearson framework is considered. The goal is to maximize the expectation of a nonlinear function of the detection probability, integrated with respect to a given probability measure, subject to a false-alarm constraint. It is shown that each power function can be realized by a generalized Bayes rule that maximizes an integrated rejection probability with respect to a finite signed measure. For a simple null hypothesis and a composite alternative, optimal single-threshold decision rules based on an appropriately weighted likelihood ratio are derived. The analysis is extended to composite null hypotheses, including both average and worst-case false-alarm constraints, resulting in modified optimal threshold rules. Special cases involving exponential family distributions and numerical examples are provided to illustrate the theoretical results.