Equivalence testing with data-dependent and post-hoc equivalence margins

TL;DR

This paper proposes a data-dependent, probabilistically bounded equivalence margin for testing effects, offering more practical and flexible decision-making tools than traditional fixed-margin approaches, supported by e-value methods.

Contribution

It introduces a novel paradigm of reporting data-dependent equivalence margins with probabilistic guarantees, generalizes to margin curves, and derives e-values for models with strict total positivity.

Findings

Data-dependent margins bound the true effect with high probability.

The approach generalizes to margin curves under post-hoc selection.

E-values are derived for models with strict total positivity.

Abstract

Equivalence testing compares the hypothesis that an effect $\mu$ is large against the alternative that it is negligible. Here, `large' is classically expressed as being larger than some `equivalence margin' $\Delta$. A longstanding problem is that this margin must be specified but can rarely be objectively justified in practice. We lay the foundation for an alternative paradigm, arguing to instead report a data-dependent margin $\widehat{\Delta}_\alpha$ that bounds the true effect $\mu$ with probability $1 - \alpha$. Our key argument is that $\widehat{\Delta}_\alpha$ is more useful than a test outcome at a fixed margin $\Delta$, as measured by the guarantees it offers to decision makers. We generalize this to a curve of margins $\alpha \mapsto \widehat{\Delta}_\alpha$, uniformly valid under the post-hoc selection of the margin. These ideas rely on e-values, which we derive for models that are strictly totally positive of order 3, nesting the classical z-test and t-test settings.