Testing for replicability in a follow-up study when the primary study hypotheses are two-sided

TL;DR

This paper proves that existing methods for controlling false replicability claims are valid when testing for replication of two-sided hypotheses in follow-up studies, by using primary study directions to guide one-sided testing.

Contribution

It demonstrates the validity of applying Heller, Bogomolov, and Benjamini's methods with primary study directions to control error rates in two-sided hypothesis replication.

Findings

Methods achieve directional control over error measures.

Validation of existing procedures for two-sided hypotheses.

Theoretical proof of error control validity.

Abstract

When testing for replication of results from a primary study with two-sided hypotheses in a follow-up study, we are usually interested in discovering the features with discoveries in the same direction in the two studies. The direction of testing in the follow-up study for each feature can therefore be decided by the primary study. We prove that in this case the methods suggested in Heller, Bogomolov, and Benjamini (2014) for control over false replicability claims are valid. Specifically, we prove that if we input into the procedures in Heller, Bogomolov, and Benjamini (2014) the one-sided p-values in the directions favoured by the primary study, then we achieve directional control over the desired error measure (family-wise error rate or false discovery rate).