An Epidemiologic Approach for Estimating Risk Reduction and Asymptotic Power on the Log-Difference Scale

TL;DR

This paper explains a statistical method to compare the effectiveness or harm of two groups using a common reference group, helping to reduce bias in studies.

Contribution

The paper introduces a framework for estimating risk reduction and test power using a log-difference scale with a common control group.

Findings

A large-sample framework for conditionally independent comparisons is reviewed.

Methods for estimating test power based on sample size are demonstrated.

Abstract

When comparing the efficacy or harmfulness of two groups (e.g., drugs, devices, assays, interventions, environmental toxins), it is important to minimize bias by making this comparison with respect to a common referent-control group, assuming random allocation. Under such a scenario, one can estimate risk reduction for a new therapy on a log-difference, relative effect scale. The current manuscript reviews the large-sample framework for this conditionally independent comparison and demonstrates how to estimate test power for a given sample size.

Figures

Full text

1. Introduction

A frequently used clinical trial design is to directly compare a new treatment against a standard therapeutic compound. However, many years may have passed since the standard compound was originally approved. During this time, variations in the disease process, manufacturing changes, or practice deviations may have occurred, such that the standard treatment is no longer efficacious, or is effective to a much lesser degree than originally marketed.

Observing a statistically significant absolute effect (risk) reduction (i.e., the standard minus new therapy event rate) may not necessarily prove the clinical usefulness of the new treatment unless the clinical trial design includes a common comparison group as a referent-control. This contrast allows for the side-by-side evaluation of the new and standard agents in the context of relative effect reduction on the log-difference scale. Importantly, the use of a shared referent-control arm minimizes bias and threats to internal validity compared with assessing the absolute risk difference.

Power and sample size are important components in the planning of a clinical trial. However, the literature is largely silent on the topic of a conditionally independent shared comparison arm when assessing risk reduction on the log-difference, relative effect scale.

2. Preliminaries

2.1. Conditionally Independent, Large-Sample Distribution

Consider a test to compare two relative effect estimates (REEs), denoted as $[eqn]$ on the log-difference scale, with respect to a common referent-control group. Approximate log-normally distributed REEs include relative risks (RR), odds ratios (OR), and hazard ratios (HR). A variance-stabilizing, logarithmic transformation tends to make the difference of REEs closer to a normal distribution by reducing any potential relationship between the variance and mean. The application of this transformation is especially useful when dealing with probabilities near 0 or 1.

Assuming that (a) the standard error ( $[eqn]$ ) for the logarithm of $[eqn]$ does not differ considerably from that for $[eqn]$ , (b) corresponding sample sizes $[eqn]$ are reasonably large, (c) $[eqn]$ and $[eqn]$ with a common referent-control arm are asymptotically consistent, but not necessarily unbiased, estimates of the true parameters, and (d) the large-sample distribution for the logarithm of $[eqn]$ is approximately Gaussian, with estimated event rates for both groups not being too close to 0 or 1, it follows from the Le Cam–Lévy–Ferguson “Conditional Martingale Limit Theorem” that we obtain the following equation:

$$[eqn]$$

where $[eqn]$ represents the expectation, and $[eqn]$ is the standard normal distribution [1]. The single tilde, “~”, indicates asymptotic equivalence of the same magnitude, under the assumption that the underlying Martingale processes are stationary, ergodic, and F-measurable [2], with finite variance.

The cumulative distribution function for $[eqn]$ divided by that for a standard normal curve approaches unity for large sample sizes. Under asymptotic regularity conditions, $[eqn]$ is conditionally independent of $[eqn]$ , given a common referent-control group. That is, the probability (A) for an observation in the numerator of $[eqn]$ is not affected by the probability (B) for an observation in the numerator of $[eqn]$ given the probability (C) for a shared observation in the denominators of $[eqn]$ Mathematically stated, it follows that ( $[eqn]$ ⫫ $[eqn]$ is equivalent to ( $[eqn]$ ⫫ $[eqn]$

The denominator of the limit described above, denoted as $[eqn]$ yields the square root of the sample variance estimate of $[eqn]$ which may be approximated as follows [3]:

$$[eqn]$$

2.2. Confidence Interval Method for Estimating the Sample Variance

Lower and upper $[eqn]$ –level confidence intervals (CIs) for the indicated REEs are given as $[eqn]$ and $[eqn]$ . That is, for $[eqn]$ we obtain the following expression:

$$[eqn]$$

where $[eqn]$ refers to the probability of rejecting the null hypothesis when it is true, and $[eqn]$ corresponds to the $[eqn]$ percentile of $[eqn]$ [e.g., $[eqn]$ ]. Using the above bounds and rearranging, it readily follows that we obtain the following equation:

$$[eqn]$$

2.3. Alternative Computational Formula for SElogRR^

Let $[eqn]$ denote the $[eqn]$ cell sizes for a contingency table with corresponding probabilities of $[eqn]$ . Here, the row exposures and column outcomes are indexed by $[eqn]$ and $[eqn]$ The relative risk estimate $[eqn]$ also known as the risk ratio, is defined as follows:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ follow a binomial distribution. By applying the delta method (Taylor series approximation for large sample sizes), we obtain the following equations:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

Accordingly, we obtain the following equation:

$$[eqn]$$

The advantage of this direct formula is the increased computational speed gained by foregoing the CI method for estimating $[eqn]$ . In the case of ORs and HRs, one can derive similar computational approximations using the delta method.

2.4. Percentage Relative Effect (Risk) Reduction

When $[eqn]$ , the percentage relative effect reduction $[eqn]$ is computed as $[eqn]$ and vice versa when $[eqn]$ i.e., $[eqn]$ If $[eqn]$ , a null result is obtained for the $[eqn]$ For example, if $[eqn]$ and $[eqn]$ , then $[eqn]$ = $[eqn]$ .

2.5. Null Hypothesis and p-Value

In the case of a null hypothesis (H_0_), $[eqn]$ are deemed to be equal, versus the alterative scenario in which they differ [3]. That is, H_0_: $[eqn]$ which is equivalent to a null $[eqn]$ compared with H_1_: $[eqn]$ In the case of (H_0_), both $[eqn]$   $[eqn]$ equal zero.

A p-value for the log-difference of two REEs is estimated as follows:

$$[eqn]$$

where

$$[eqn]$$

and

$$[eqn]$$

2.6. Confidence Interval Decision Criteria

The disjoint CI percentage corresponding to an equivalent $[eqn]$ -level test is computed as follows:

$$[eqn]$$

This yields slightly wider regions to compensate for the pooled (versus non-pooled) sample variance [4]. For example, given that $[eqn]$ = 0.01, 0.05, and 0.10, the difference between the estimates $[eqn]$ is declared as statistically significant if the 93.1452%, 83.4224%, and 75.5206% CIs for $[eqn]$ do not overlap.

Analogously, the difference between the estimates $[eqn]$ may be declared as statistically significant when the following equation excludes unity:

$$[eqn]$$

3. Power and Sample Size

3.1. Power

The power of a study indicates how frequently a statistical test will detect the falsehood of an underlying null hypothesis when it is false (i.e., $[eqn]$ . Let $[eqn]$ denote the probability of failing to reject the null the hypothesis when it is false (i.e., $[eqn]$ ). It follows that we obtain the following equation:

$$[eqn]$$

The area to the left of $[eqn]$ under a standard normal distribution yields the desired power at the $[eqn]$ -level of statistical significance for a two-sided test. When use of a one-tailed test is desired, $[eqn]$ , rather than $[eqn]$ , is used in this formula.

3.2. Sample Size and Variability

The confidence interval method may be used to approximate the standard error for the logarithm of the respective relative effect estimates ( $[eqn]$ ) corresponding to an initial rxc (row by column) = (n) contingency table, where (n) denotes the total sample size (i.e., combined rows). Cell frequencies of the pilot sample table are increased multiplicatively (in an iterative fashion) to obtain the corresponding standard errors for different sample sizes and corresponding power.

Using this method, it is important that the initial 3 × 2 contingency table be sufficiently large to accurately estimate the sample variability within a specified fraction of the population variance [5]. As a rule of thumb, the proportional width of the α-level CI for the population variance of a continuous variable is defined as follows:

$$[eqn]$$

This value should be as large as the sample variance, where $[eqn]$ is the $[eqn]$ percentile of the chi-squared distribution with $[eqn]$ degrees of freedom. When α = 0.05, this corresponds to a minimum sample size of 38. However, in practice, a more conservative sample size of 90 (30 per treatment row) is typically chosen as the starting point to account for the binary outcomes.

3.3. Simulation of Observations from a Multinomial Distribution

Given (n) distinct trials, the probability that a mutually exclusive set of ( $[eqn]$ ) non-negative random variates ( $[eqn]$ ) takes on a particular value ( $[eqn]$ ) is given as follows:

$$[eqn]$$

where

$$[eqn]$$

This probability is known as a multinomial distribution because of the following expression:

$$[eqn]$$

where the latter is a multinomial series.

The multinomial distribution, as illustrated in Example 2, plays an important role in validating the asymptotic normality of the conditional test statistic.

4. Computational Methods

Analyses were performed and validated in SAS 9.4 (Cary, NC).

5. Example 1—Comparison of Potassium and Sodium Salts

Potassium salt substitutes can be helpful for lowering sodium intake and controlling high blood pressure. Among a cohort of borderline hypertensive (but otherwise healthy) patients, a team of clinical epidemiologists aimed to determine if randomization to a potassium salt substitute (1500 mg/d) reduces the 24-month risk of major adverse cardiovascular outcomes and non-cancer death compared with a formulary of sodium salt (1500 mg/d), with a common referent-control arm of 2300 mg/d sodium salt.

In a 1:1:1 pilot clinical trial, a 9.52% relative effect (risk) reduction on the log-difference scale was observed at 24 months post-randomization (Table 1). Based on this promising result, the team decided to conduct a larger population-based phase III clinical trial. To achieve at least 80% power for a similar %RLD (discounting dropouts), the new pivotal trial would need to randomize 36,835 patients per study arm at the α = 0.05 level of statistical significance (Table 2; see Appendix A for SAS code).

6. Example 2—Monte Carlo Simulation of the Conditional Martingale Limit Theorem

While a theoretical basis exists for the asymptotic normality and conditional independence of the test statistic [1,2], Monte Carlo methods can be used to validate the large- sample properties of this method.

A total of 1,000,000 trials, repeated for each sample of 900,000 patients (300,000 per arm), were drawn from a multinomial distribution using the cell probabilities given in Example 1 (with T_0_ denoting the common referent-control group). A standardized value was computed for each simulated observation for $[eqn]$ by subtracting $[eqn]$ from this estimate and dividing it by $[eqn]$ . Plotting these values as a histogram yields Figure 1. The corresponding statistics (mean = 0, standard deviation = 1, and approximate zero values for skewness and kurtosis) indicate that the z-scores follow a standard normal distribution, as visually confirmed by the Normal (0,1) curve overlay. Furthermore, normality was not rejected by the Kolmogorov–Smirnov, Cramer–von Mises, or Anderson–Darling tests (see Appendix B for SAS code).

For large samples, this example supports the use of normal-theory methods to estimate power for a 1:1:1 randomized design with a common referent-control arm. In effect, the covariance of $[eqn]$ can be disregarded as the sample size increases toward infinity, allowing one to assume an independent, identically distributed (i.i.d.) standard normal test statistic.

7. Discussion

7.1. Overview

The comparison of two drugs with respect to a common comparator (referent-control) group is a powerful and efficient approach for conducting a clinical trial. Importantly, this method preserves the randomization of the originally assigned patient groups [6]. Additionally, it allows for determining risk reduction on the log-difference scale. The current manuscript presents a novel, computer-based technique for estimating the power of the latter by multiplicatively increasing the cell sizes of an initial pilot sample. This yields the inflated variances needed to compute power for larger sample sizes while still allowing the use of traditional normal-theory methods. As reasoned below, the common referent-control design does not require multiplicity adjustment.

7.2. Multiplicity Adjustment

The family-wise type I error rate, which is the probability of at least one false positive test among all hypotheses being tested, is not increased with respect to a common referent-control group [7]. The associated false positive comparison-wise error rate at the individual level is also not inflated, as the “expected proportion of incorrectly rejected hypotheses will not exceed the significance level used in the individual tests” [7,8].

While a treatment is the “family” over which one typically judges the need for multiplicity, there is indistinctness regarding how this entity is defined. Multiplicity control is not required if the underlying therapies have distinct mechanisms of action [9]. That is, “if each decision to reject each individual null hypothesis depends on no more than one significance test, then none of the individual tests constitute a family”, such that an $[eqn]$ -level adjustment for any single hypothesis is unnecessary (except if one performs “disjunction testing of a joint intersection null hypothesis”) [10]. While cases exist wherein “the decision on the omnibus null will seem to contradict decisions on individual nulls”, it is important to remember that “rejecting or not rejecting a null is not certain proof that the null is false or true” [10]. Therefore, multiplicity correction for CIs) is needed when the omnibus null can only be evaluated through “a set of surrogate nulls” and cannot be “generalized to testing any diversity of unconnected nulls over the course of a study” [11].

Adjustment is counterintuitive in the case of conjunctive (intersection–union) testing, in which all tests must be significant for the joint null hypothesis to be rejected [9]. However, multiplicity adjustment for CIs is advisable if the comparative therapies are related, as is the case when evaluating “different dosages or regimens of a treatment compared with the same control arm” [12]. This includes trials with collective conditions “if there are no less than two primary hypotheses, unless one assumes that there is an explicit hierarchy in the multiple hypotheses” [12].

Multiplicity adjustment of CIs in the denominator of a relative-effects (log-difference) test of two compounds is generally not performed in practice, as this is equivalent to a stand-alone interaction test with a pooled variance structure [3]. If requested by a regulatory agency, the intervals can be easily adjusted for multiplicity using the Hochberg step-up procedure [13]. However, when “such adjustments are applied unnecessarily, potentially effective treatments may be discarded prematurely” [9].

7.3. Limitations

A limitation of REEs is that they are ratios and tend to have a slightly skewed log-normal distribution, which creates issues when the parameters of a test statistic are estimated from the sample. According to Jensen’s inequality, the expectation will be less than or equal to its true value. To some degree, this bias is an advantage because it helps to offset the Cauchy infinite variance problem that may result from taking the ratio of two normally distributed variables [1]. Barring extreme degenerate examples, it is noted that $[eqn]$ asymptotically assumes a moderately well-behaved, normal shape in the limit.

The literature offers little consistent advice regarding the lower limit for the sample size needed to satisfy the underlying large-sample assumptions of the test statistic. It is also unclear what level of inaccuracy is incurred when estimating a z-score test statistic when the sample size is small and the observations are not fully independent. By treating the logarithmic transformation of $[eqn]$ as a mean difference and applying Gauss’ hypergeometric series, one may heuristically conclude that the small-sample test statistic follows a pseudo Behrens–Fisher distribution with $[eqn]$ degrees of freedom [14], as follows:

$$[eqn]$$

An alternative, albeit lower-power, approach is the Lepage test, which treats the problem as a nonparametric test for central tendency and dispersion [15]. In certain situations, nonparametric methods are better suited (less sensitive) for handling violations of independence.

Addressing the exact convergence rates for a conditional Martingale-type central limit theorem, which are ostensibly slower than $[eqn]$ is a complex topic beyond the scope of the current manuscript and is best deferred to future research [16]. Nonetheless, it is common practice in many applied scenarios to accept that the underlying data are “close enough” to normal to move forward with normal-theory methods.

Imprecise rounding is also a concern. As REEs are expressed on a logarithmic scale, with significant departures from linearity occurring near unity, it is best to avoid intermediate rounding when computing the test statistic. Fortunately, this rarely poses a problem when the operations are performed by a computer rather than by manual means.

In theory, power for a given sample size may be estimated by simulating datasets from a conditional multinomial distribution, with prescribed cell probabilities being reflective of the underlying hypothesized parameters. Power is then defined as the proportion of the simulated test statistics that fall within the “a priori” alpha rejection region. However, the simulation method is computationally intensive and introduces uncertainty in the values at the distribution tails. As large sample sizes are often needed to detect small effect sizes for risk reduction, the process may be resource-prohibitive or prone to computational errors, in which case normal-theory methods may be the better choice for computing power. While robust estimation methods are available in most standard statistical packages to simulate the covariance of REEs and corresponding test statistics involving correlated data, boundary issues regularly result in algorithmic non-convergence. The simulation of power also is problematic for heavy-tailed, Cauchy-like distributions, which may require sample sizes on the order of millions to properly characterize outlying values. Furthermore, multinomial-type power computations are predicated on the observed pilot sample effect size. Thus, simulation cannot be used to determine power for a range of minimally detectable $[eqn]$ ) values based on a single effect size estimate. In such cases, separate pilot samples will be needed to determine the desired range of values.

7.4. Future Directions

Future investigations are directed at estimating power when sample sizes are small and the assumption of conditional independence is questionable. This will include cases where it is difficult to establish the consistency, stability, and normality of underlying data estimates and parameters. Innovative research efforts focusing on nonparametric, two-sample conditional tests and Bayesian methods also may be useful [17,18].

8. Conclusions

The use of a common reference-control arm is an important tool for minimizing bias in a randomized clinical trial. Assuming conditional independence, a large-sample method is presented to estimate power for risk reduction on the log-difference scale. This computationally straightforward approach offsets the uncertainty and potential bias associated with tests of absolute risk reduction.