Numerical Analysis of Test Optimality

TL;DR

This paper introduces a numerical framework to evaluate whether ad hoc tests in complex testing environments are nearly optimal by approximating their power relative to a theoretical power envelope, with practical applications demonstrated.

Contribution

The paper develops a nested optimization approach to assess and approximate the optimality of ad hoc tests in nonstandard testing scenarios.

Findings

Rejection probabilities form an approximate power envelope.

Method successfully applied to weak instrument-robust tests.

Provides convergence guarantees and practical implementation insights.

Abstract

In nonstandard testing environments, researchers often derive ad hoc tests with correct (asymptotic) size, but their optimality properties are typically unknown a priori and difficult to assess. This paper develops a numerical framework for determining whether an ad hoc test is effectively optimal - approximately maximizing a weighted average power criterion for some weights over the alternative and attaining a power envelope generated by a single weighted average power-maximizing test. Our approach uses nested optimization algorithms to approximate the weight function that makes an ad hoc test's weighted average power as close as possible to that of a true weighted average power-maximizing test, and we show the surprising result that the rejection probabilities corresponding to the latter form an approximate power envelope for the former. We provide convergence guarantees, discuss practical implementation and apply the method to the weak instrument-robust conditional likelihood ratio test and a recently-proposed test for when a nuisance parameter may be on or near its boundary.