Blinded sample size re-estimation in equivalence testing

TL;DR

This paper examines how blinded sample size re-estimation can lead to increased type I error rates in equivalence testing, especially with small sample sizes, and provides theoretical and simulation insights.

Contribution

It explains why type I error inflation occurs in equivalence testing during blinded sample size re-assessment and quantifies this effect through simulations and theory.

Findings

Type I error violations are more pronounced in equivalence testing than superiority testing.

Small sample sizes during blinded re-assessment lead to significant error inflation.

The paper provides theoretical derivations and simulation results quantifying error inflation.

Abstract

This paper investigates type I error violations that occur when blinded sample size reviews are applied in equivalence testing. We give a derivation which explains why such violations are more pronounced in equivalence testing than in the case of superiority testing. In addition, the amount of type I error inflation is quantified by simulation as well as by some theoretical considerations. Non-negligible type I error violations arise when blinded interim re-assessments of sample sizes are performed particularly if sample sizes are small, but within the range of what is practically relevant.

Tables

Table 1. Table 1: Simulation settings

Parameter	Values
$\tilde{n}$	10, 15, 20, 25, 30, 40, 50, 60
$n_{min}/\tilde{n}$	1, 1.2, 1.4, 1.6, 1.8, 2
$n_{max}/\tilde{n}$	2, 2.5, 3, 3.5, 4, infinity
${\delta }_0/\sigma ={\delta }_0$	0.05 to 1.5 at 0.05 interval

Table 2. Table 2: Peak α 𝛼 \alpha and its corresponding effect size

$\tilde{n}$	$\%$ Case 1	${\delta }_0/\sigma$
10	6.26	1.20
15	5.78	0.95
20	5.63	0.85
25	5.55	0.80
30	5.45	0.75
40	5.34	0.60
50	5.30	0.60
60	5.23	0.55
80	5.18	0.45

Equations

$${\tilde{n}}_j$$

$$m_j$$

$$n_j$$

$$n_{j,Min}$$

$$n_{j,Max}$$

$${\hat{\sigma }}_T^2=\frac{Q}{{\tilde{n}}_{\ast }-1}=\frac{1}{{\tilde{n}}_{\ast }-1}\sum _{j=1}^2\sum _{i=1}^{{\tilde{n}}_j}{\left(y_{ij}-\bar{y}\right)}^2,$$

$$Q_1$$

$$Q_2$$

$$E(Q)=E(Q_{1})+E(Q_{2})=(\tilde{n}_{*}-2)\sigma^{2}+\bigg{(}1+\frac{\tilde{n}_{1}\tilde{n}_{2}}{\tilde{n}_{*}}\frac{\delta^{2}}{\sigma^{2}}\bigg{)}\sigma^{2},$$

$$E({\hat{\sigma }}_T^2)=\frac{1}{{\tilde{n}}_{\ast }-1}E(Q)={\sigma }^2\left(1+\frac{{\tilde{n}}_1{\tilde{n}}_2}{{\tilde{n}}_{\ast }({\tilde{n}}_{\ast }-1)}\frac{{\delta }^2}{{\sigma }^2}\right).$$

$${\text{H}}_{01}:\delta \le {\delta }_{low}\text{vs.}{\text{H}}_{A1}:\delta >{\delta }_{low},\text{ and}$$

$${\text{H}}_{02}:\delta \ge {\delta }_{up}\text{vs.}{\text{H}}_{A2}:\delta <{\delta }_{up},$$

$$\sqrt{\frac{n_1{n}_2}{n_1+n_2}}\cdot \frac{d-{\delta }_{low}}{s}>t_{1-\alpha }(n_1+n_2-2),$$

$$\sqrt{\frac{n_1{n}_2}{n_1+n_2}}\cdot \frac{d-{\delta }_{up}}{s}<t_{\alpha }(n_1+n_2-2),$$

$$s^2=\frac{1}{n_1+n_2-2}\sum _{i=1}^2\sum _{j=1}^{n_i}(y_{ij}-\overline{y_i}{)}^2.$$

$$\hat{N}=\frac{2(Z_{1-\beta /2}+Z_{1-\alpha }{)}^2}{(({\delta }_0-D)/{\hat{\sigma }}_T{)}^2},$$

$$Var({\hat{\sigma }}_T^2)=\frac{2{\sigma }^2}{(2\tilde{n}-1{)}^2}\left[\tilde{n}(2+{\delta }^2)-1\right]=\mathcal{O}\left(\frac{{\delta }^2}{\tilde{n}}\right).$$

$$E(\hat{N})=2(Z_{1-\beta /2}+Z_{1-\alpha }{)}^2\left(\frac{1}{{\delta }_0^2}+\frac{\tilde{n}}{4\tilde{n}-2}\right),$$

$$SD(\hat{N})=2(Z_{1-\beta /2}+Z_{1-\alpha }{)}^2\times \frac{SD({\hat{\sigma }}_T^2)}{{\delta }_0^2}=\mathcal{O}\left(\frac{1}{{\delta }_0}\cdot \frac{1}{\sqrt{\tilde{n}}}\right).$$

$$\underset{{\delta }_0\to 0}{\lim }\frac{E(\hat{N})}{SD(\hat{N})}=\frac{1}{{\delta }_0}=\infty .$$

$$\underset{{\delta }_0\to \infty }{\lim }\hat{N}=\frac{\tilde{n}}{2\tilde{n}-1}{\left(Z_{1-\beta /2}+Z_{1-\alpha }\right)}^2,$$

$$\underset{{\delta }_0\to \infty }{\lim }SD(\hat{N})=0.$$

$$H_{02}:\delta \ge {\delta }_{up}$$

$$t_{up}=\sqrt{\frac{n_1}{2}}\cdot \frac{d-{\delta }_{up}}{\sqrt{\frac{Q_1}{n-2}}}\le t_{\alpha }(n-2).$$

$$\displaystyle P\left(t_{up}\leq t_{\alpha}(n-2)\ \big{|}\ Q_{1}=x\right)=$$

$$P\left(\sqrt{\frac{n_1}{2}}\cdot (d-{\delta }_{up})\le q\cdot \sqrt{\frac{x}{n-2}}\right)=\Phi \left(q\cdot \sqrt{\frac{x}{n-2}}-\sqrt{\frac{n_1}{2}}(\delta -{\delta }_{up})\right),$$

$$\displaystyle P\left(t_{up}\leq t_{\alpha}(n-2),\ \sqrt{\frac{n_{1}}{2}}\left|d\right|\leq\sqrt{c-x}\ \bigg{|}\ Q_{1}=x\right)=$$

$$P(\sqrt{\frac{n_1}{2}}\cdot (d-{\delta }_{up})\le q\cdot \sqrt{\frac{x}{n-2}},\sqrt{\frac{n_1}{2}}\cdot (d-{\delta }_{up})\le \sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up},$$

$$\sqrt{\frac{n_1}{2}}\cdot (d-{\delta }_{up})\ge -\sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up})=$$

$$\Phi \left(\min \left(\sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up},q\cdot \sqrt{\frac{x}{n-2}}\right)\right)-\Phi \left(-\sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up}\right)$$

$$P\left(t_{up}\le t_{\alpha }(n-2),Q_1+Q_2\le c\right)=$$

$$\displaystyle\quad\int_{x=l^{*}}^{c^{*}}\bigg{\{}\Phi\left(\min\left(\sqrt{c-x}-\sqrt{\frac{n_{1}}{2}}\delta_{up},\ q\cdot\sqrt{\frac{x}{n-2}}\ \right)\right)$$

$$\displaystyle\qquad-\Phi\left(-\sqrt{c-x}-\sqrt{\frac{n_{1}}{2}}\delta_{up}\right)\bigg{\}}\cdot f(x)dx,$$

$$q\cdot \sqrt{\frac{x}{n-2}}=-\sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up}$$

$$q\cdot \sqrt{\frac{x}{n-2}}=\sqrt{c-x}-\sqrt{\frac{n_1}{2}}{\delta }_{up}$$

$$P\left(t_{up}\le t_{\alpha }(n-2),Q_1+Q_2\le c\right)\cdot {\left(P\left(Q_1+Q_2\le c\right)\right)}^{-1}.$$

Full text

Keywords: bioequivalence, biosimilar, non-inferiority, sample size re-estimation, TOST, type I error control

1 Introduction

Discussions on sample sizes re-estimation (SSR) have appeared in statistical literature as early as the 1940’s [1]. Since then a growing body of literature has emerged. For the two-sample $t$-test of a new treatment versus a control, methods based on estimators of the sample variance which can be calculated without unblinding the group affiliation of the observations available at an interim analysis have been introduced [2, 3]. The simplest of these is the “total variance estimate” which is the usual one-sample variance estimate calculated from treating all observations as if they were from only one treatment group. Although it has been shown that under this approach the type I error is not strictly controlled [4, 5], in the case of superiority testing, the type I error violations are extremely small and even then occur only in small samples. From an applications perspective, these investigations therefore reinforce the recommendations made by [2] and [3]. However, in the case of non-inferiority (NI) testing, and thus also in equivalence (EQ) testing, this inflation can be larger [5, 6, 7, 8]. In this paper we take a closer look at the non-trivial type I error inflation in NI and EQ testing and the factors that impact it. We also provide practical recommendations to clinicians and data analysts who wish to implement a blinded SSR in their equivalence studies.

We base our discussion on a design with two stages and two treatment groups. Stage 1 consists of subjects whose data are used for estimating the total variance; stage 2 consists of subjects who are additionally recruited after the interim. In certain cases, no additional subjects are needed and stage 2 does not have any subjects. An important assumption is that beside the total sample size, other elements of the study design do not change after the interim.

We use the following notations in our discussions. For treatment group $j,\ j=\{1,\ 2\}$:

[TABLE]

We assume $n_{j,Max}\geq n_{j,Min}\geq\tilde{n}_{j}$.

In Section 2, we derive a decomposition of the total variance observed at the time of blinded SSR to demonstrate how type I error inflation arises in NI and EQ tests. Section 3 provides a brief introduction to the two one-sided tests (TOST) approach to EQ testings. Section 4 outlines the simulation settings and presents the results. Section 5 discusses how different factors impact the type I error. In Section 6 we summarize our findings.

2 Type I error violation in non-inferiority testing

A common strategy for EQ testing is the TOST approach [9]. This uses two NI tests. For this reason we start by looking at the latter.

Assume that we want to test $H_{0}:\delta\geq\delta_{0}$ versus $H_{A}:\delta<\delta_{0}$ at level $\alpha$ on independent normal observations $y_{ij}$ from $N(\mu_{j},\sigma^{2})$ in two groups $j=1,2$ and $i=1,\ \dots\ n_{j}$. Let $\delta=\mu_{1}-\mu_{2}$, and $\bar{y}_{j}$ be the mean of the observations in group $j$. Without loss of generality, assume furthermore that $\delta_{0}>0$. We proceed with the “total variance”

[TABLE]

where $\tilde{n}_{*}=\tilde{n}_{1}+\tilde{n}_{2}$, and $\bar{y}=\frac{\tilde{n}_{1}\bar{y}_{1}+\tilde{n}_{2}\bar{y}_{2}}{\tilde{n}_{*}}$ is the mean across the two groups.

To calculate $E(\hat{\sigma}_{T}^{2})$, we decompose $Q$ into $Q_{1}$ and $Q_{2}$, where

[TABLE]

It follows that $Q=Q_{1}+Q_{2}$. By Cochran’s theorem, $Q_{1}$ and $Q_{2}$ are stochastically independent with distributions $Q_{1}\sim\sigma^{2}\chi^{2}(\tilde{n}_{*}-2)$ and $Q_{2}\sim\sigma^{2}\chi^{2}(1;\frac{\tilde{n}_{1}\tilde{n}_{2}}{\tilde{n}_{*}}\frac{\delta^{2}}{\sigma^{2}})$, which is a non-central $\chi^{2}$ distribution. Hence,

[TABLE]

and

[TABLE]

To illustrate how type I error inflation may arise, consider the following SSR rule:

If $\hat{\sigma}_{T}^{2}$ is large, i.e. $\hat{\sigma}_{T}^{2}>c$ for some preselected threshold $c>0$, we recruit $m_{j}$, $j=1,2$, additional subjects. 2. 2.

If $\hat{\sigma}_{T}^{2}$ is small, i.e. $\hat{\sigma}_{T}^{2}\leq c$, we do not recruit additional subjects, but stop and test right away with the $n=n_{1}+n_{2}=\tilde{n}_{1}+\tilde{n}_{2}$ subjects we have.

If the true $\delta>0$, we can see from Equation (2) that a small value of $\hat{\sigma}_{T}^{2}$ provides evidence against $H_{0}$, because $E(\hat{\sigma}_{T}^{2})$ does not only depend on the variance, but also on the squared true mean difference between the two groups. Hence, the rule above stops recruiting and uses the data obtained up to the interim analysis undiluted by additional observations if it happened to be in favor of $H_{A}$, but dilutes this evidence by recruiting more subjects if the data obtained up to the interim analysis shows evidence in favor of $H_{0}$. It is clear that this must inflate the type I error of the entire strategy. The same is true for other blinded sample size re-estimation rules where the final total sample size is an increasing function of $\hat{\sigma}_{T}^{2}$. On the other hand, if the sample size re-estimation rule were a decreasing function of $\hat{\sigma}_{T}^{2}$, then we would have $\alpha$ deflation.

Two remarks are important. Firstly, from Equation (2) we can see that the inflation is primarily associated with the shifted hypothesis $H_{0}:\delta\geq\delta_{0}>0$. If $\delta_{0}=0$, then at the point of $\delta=\delta_{0}$, $\hat{\sigma}_{T}^{2}$ is an unbiased estimate of $\sigma^{2}$. Regarding type I error control, what matters is the behavior of the test at $\delta=\delta_{0}$. Hence, the issue investigated here does not occur with superiority testing where $H_{0}:\delta=\delta_{0}=0$. Intuitively, a large $\hat{\sigma}_{T}^{2}$ does not provide evidence in favor of, or against, the superiority hypothesis, because a large observed $\hat{\sigma}_{T}^{2}$ can equally likely arise from $\delta$ or $-\delta$ being true. With the shifted $\delta=\delta_{0}>0$, however, a large value of $\hat{\sigma}_{T}^{2}$ will much more likely occur with a value of $\bar{y}_{1}-\bar{y}_{2}>\delta$ than with $\bar{y}_{1}-\bar{y}_{2}<-\delta$. We note in passing that simply subtracting a constant, i.e. a hypothetical true difference, from the total variance cannot influence this issue.

Secondly, the statements we have made here are qualitative: There will be a type I error inflation, but its magnitude depends on many things: the effect size $\delta_{0}/\sigma$, the stage 1 sample size, the range of possible values of the stage 2 sample size, the actual small sample properties of a test using a test statistic which is only asymptotically Normal, and the like. Equivalence testing with TOST is a case in point. If the equivalence margin is symmetric (i.e. $\delta_{0}=\delta_{up}=-\delta_{low}$), then exactly the same mechanisms are at work at both the upper and the lower end of the equivalence range, and both one-sided level-$\alpha$-tests are subject to type I error inflation with the sample size review strategy described above. The actual probability of rejection in the TOST, however, is also subject to the inherent conservatism of the TOST strategy which arises from the fact that there is a positive probability of not rejecting $H_{01}:\delta\leq-\delta_{0}$ when in fact $H_{02}:\delta\geq\delta_{0}$ is true. This conservatism of course remains and may counteract against the liberality caused by the phenomenon we discussed.

An analytical calculation of the exact type I error rate is possible, however, given the many factors at play, the derivation is often algebraically consuming. In Appendix A, we discuss the quantification of the type I error inflation in a specific case. For general discussion, we use simulation studies.

3 Equivalence testing and TOST

In the TOST approach to equivalence testing, two hypotheses are tested simultaneously:

[TABLE]

[TABLE]

where $\delta_{low}$ and $\delta_{up}$ are the lower and the upper equivalence margins. To claim equivalence, both null hypotheses must be rejected simultaneously at level $\alpha$. Using the same notations as in Section 2, and with the given reference range $(\delta_{low},\ \delta_{up})$, we reject the null hypothesis of non-equivalence $H_{0}=H_{01}\cup H_{02}$ if

[TABLE]

and

[TABLE]

where $d=\bar{y}_{1}-\bar{y}_{2}$ and

[TABLE]

This decision rule can also be described via a $(1-2\alpha)$ level confidence interval (CI) for $\delta$. If this CI is completely contained within $(\delta_{low},\ \delta_{up})$, then equivalence between the two groups is declared at level $\alpha$.

In Figure 1 we distinguish four possible outcomes with regard to the rejection of the two null hypotheses, and illustrates them in terms of the CI for $\delta$. Per these definitions, Case 1 is where equivalence is established.

Consider an EQ test using TOST where $H_{02}$ is true. Case 1 represents a false rejection and establishes an equivalence incorrectly, i.e. a type I error. Although Case 2 is the least desirable outcome, it makes TOST conservative: it falsely rejects $H_{02}$, but also falsely does not reject $H_{01}$, thus comes to the correct conclusion that the equivalence cannot be established, even though the reasoning is wrong. Case 3 is the most desired outcome since it correctly supports non-rejection of $H_{02}$ but rejection of $H_{01}$. Case 4 does not reject either null hypothesis, therefore, reaches the correct decision of not rejecting $H_{02}$.

For NI testing where $H_{02}$ is true, Case 1 and Case 2 together represent false rejection of the null hypothesis.

4 Simulation studies

We investigate the impact of stage 1 sample size ($\tilde{n}_{j}$), stage 2 sample size ($m_{j}$), and effect size on type I error inflation in a blinded SSR for EQ testing using simulation studies. Without loss of generality, we set $\sigma$ to 1, and assume symmetric equivalence margins where $\delta_{up}=\delta_{0}=-\delta_{low}$; hence, in subsequent discussions $\delta_{0}$ and effect size $\delta_{0}/\sigma$ are numerically the same.

The sample sizes for the two groups are assumed the same at every stage, so the subscript $j$ is dropped in the discussion. We follow this strategy: before the start of the study, define $\tilde{n}$, $n_{Min}$, and $n_{Max}$ where $\tilde{n}\leq n_{Min}\leq n_{Max}$. A blinded sample size re-assessment is conducted when data from $\tilde{n}$ subjects per group become available.

4.1 The SSR rule

The total sample size needed in each group, denoted $\hat{N}$, in order to reject $H_{01}$ and $H_{02}$ for a desired power $1-\beta$ is calculated based on $\hat{\sigma}_{T}^{2}$. $\hat{N}$ is compared to $n_{Min}$ and $n_{Max}$ such that

• (1)

if $\hat{N}\leq n_{Min}$, then additional $m=n_{Min}-\tilde{n}$ subjects are recruited per group in the second stage;

• (2)

if $\hat{N}>n_{Min}$, then additional $m=\text{min}(\hat{N},n_{Max})-\tilde{n}$ subjects are recruited in the second stage.

Hypotheses $H_{01}$ and $H_{02}$ are tested with $n=\tilde{n}+m$ total subjects per group at the end of the study.

4.2 Simulation settings

All of the simulations use $\alpha=5\%$ and $\beta=10\%$. Table 1 summarizes the other conditions:

Take $\tilde{n}=20$, $n_{Min}/\tilde{n}=1.4$ (or equivalently $n_{Min}=20\times 1.4=28$), and $n_{Max}/\tilde{n}=2$ (or $n_{Max}=20\times 2=40$) as an example. Our strategy says that the interim analysis is to be conducted when data from 20 subjects per group become available. Based on the total variance estimate $\hat{\sigma}_{T}^{2}$, if the total sample size $\hat{N}$ is determined to be no more than 28 subjects per group, then additional $m=8$ subjects per group will be recruited in the second stage, making it total of $n=n_{Min}=28$. On the other hand, if the re-assessed sample size $\hat{N}$ turned out to be larger than 28, then $m=\text{min}(\hat{N},40)-20$ subjects per group are to be recruited for the second stage.

One million simulations are performed for each combination of the parameter values in Table 1. Normally distributed data are generated for the two groups with a mean difference of $\delta_{0}$. This corresponds to a true $H_{02}$. In each simulated sample $t$-test is conducted between the two groups. The percentages of each of the four cases (as illustrated in Figure 1) are summarized.

4.3 A note on sample size calculation

For each simulated sample, the total sample size $\hat{N}$ is calculated from the stage 1 data by assuming a common population variance $\sigma^{2}$ estimated by $\hat{\sigma}_{T}^{2}$. Ideally $\hat{N}$ should be calculated using non-central $t$-distribution as outlined in [9]. However, this would require iterative approximation of degrees of freedom for each single simulation. Furthermore, we focus on type I error inflation which ultimately depends on the sample size formula only via $\hat{N}$. For this purpose, it is essentially irrelevant if the $t$- or the simpler $z$-formula is used. In both cases, type I error inflations follow identical patterns. Therefore, we use the usual sample size formula based on the asymptotic normal distribution:

[TABLE]

assuming 1:1 randomization ratio. Since we are blinded at interim analysis, we do not have an estimate for $\delta$; a value $D$ must be assumed. In NI and EQ testing, it is common to assume that there is no difference between groups when doing power and sample size calculations. We follow this practice and assume $D=0$.

Although Equation (3) tends to give smaller sample sizes, its dependency on $\delta_{0}/\sigma_{T}$ is the same as its $t$-distribution based counterpart. Other components such as $\tilde{n},\ m,\ n_{Min},\ n_{Max}$, etc., only impact the amount of the inflation. As Appendix A shows, a different sample size determination rule may dictate a completely different pattern of $\alpha$ behavior. Since Equation (3) and its $t$-distribution counterpart are implemented in many statistical software for power and sample size calculations, for example NCSS PASS or R package PowerTOST [10, 11], we make this our main objective of investigation.

4.4 Results

We first present a benchmark example. Figure 2 shows the observed type I error rate for NI and EQ testings for the setting where $\tilde{n}=15$, $n_{Min}=18$, and $n_{Max}=30$. What this rule says is that we conduct the interim when there are 15 subjects in each group; if the observed total variance $\hat{\sigma}_{T}^{2}$ is small enough such that we will only need up to a total of 18 subjects per group in order to reject $H_{0}$ with 90% power at 5% nominal $\alpha$ level, then we will recruit 3 additional subjects in each group; otherwise, we will recruit however many subjects needed but not exceeding 30.

In Figure 2 there are two red dashed lines on each panel representing the 95% CI around 5% in one million simulations. Any values within these two dashed lines can be attributed to random variation. In Panel A, percentages of Case 1 are plotted against the effect size. Recall a Case 1 is where the CI for the difference between the two groups is completely contained within the equivalence margins. Its probability under $H_{0}$ is the type I error rate for an EQ testing. When effect size is at or below 0.30, the standard deviation $\sigma$, which is set to be 1 for all the simulations, is much too large for $\delta_{0}$. The interim total variance $\hat{\sigma}_{T}^{2}$, serving as an approximation for $\sigma^{2}$, will tend to be very large in this area as well. Therefore, even with the largest sample size allowed according to our SSR rule, nearly no CI for the difference between the two groups are narrow enough to be within $(-\delta_{0},\delta_{0})$. Hence, no type I error is committed under the EQ hypothesis.

However, this is not to say that $H_{02}:\delta\geq\delta_{0}$ is correctly not rejected. Panel B of Figure 2 shows the percent of Case 2 against effect size. Recall a Case 2 is where the lower end of the confidence interval is below the lower margin. The probability of false rejection of $H_{02}$ starts out at 5% and drops below that at effect size equals to 0.30 for exactly the reason that $\sigma$ is too large relative to $\delta_{0}$ in this area. If $H_{02}$ is indeed falsely rejected, the chances are higher that they are Case 2 rather than Case 1.

As $\delta_{0}$ becomes larger the CI for the group difference is becoming narrower given the same SSR rule. This in turn shows in Panel A such that the percentage of Case 1 is increasing sharply between effect size 0.30 and 0.5; at the same time, percentage of Case 2 starts to decrease, and reaching 0% at 0.55 just when percent Case 1 reaches 5%. If we look at Panel C, we see the two cases add up nicely to about 5% or exceeding that where there is inflation. In fact, Panel C (or Panel D for a close-up) are the observed $\alpha$ in a NI test against $H_{02}$ with the given SSR rule. For both NI and EQ testing, the $\alpha$ inflation happens between effect size 0.70 and 1.25 with a peak at 0.95.

The patterns in Figure 2 are consistent in all of the settings we examined. The difference is in where the peak $\alpha$ occurs, and how much it inflates to. Figure 3 summarizes the percent of Case 1 in form of heatmaps where a darker shade indicates a larger $\alpha$ value. In addition, we see how $\tilde{n}$, $n_{Min}$, and $n_{Max}$ influence $\alpha$ inflation from these plots. Four stage 1 sample sizes, $\tilde{n}=10,\ 15,\ 20,\ \text{and}\ 30$, are selected for display. They are in columns from left to right. The rows in Figure 3 represent $n_{Max}$ in terms of the ratio $n_{Max}/\tilde{n}$. The selected values are $n_{Max}/\tilde{n}=2,\ 3,\ 4,$ and infinity. Within each of the 16 heatmaps, the $x$-axis represents the effect size from 0.05 to 1.5; the $y$-axis represents the $n_{Min}$ in terms of the ratio $n_{Min}/\tilde{n}$. Note that in our simulations we have chosen discrete $n_{Min}/\tilde{n}$, and the y-axis is not truly continuous. The benchmark example we have shown in Figure 2 is highlighted with a red rectangle.

The graphs show that inflations of type I errors occur at $\delta_{0}/\sigma$ between 0.8 and 1.5, and that they are more severe with small stage 1 sample sizes and when $n_{Min}=\tilde{n}$.

5 Discussion

5.1 Stage 1 sample size and $\alpha$ inflation

In Figure 3 we see a consistent decrease of $\alpha$ inflation as $\tilde{n}$ increases. This is related to the precision in the estimate of $Var(\hat{\sigma}_{T}^{2})$. Given Equation (1), we derive

[TABLE]

With fixed $\delta$, $Var(\hat{\sigma}_{T}^{2})$ decreases as $\tilde{n}$ increases. The sample size formula in Equation (3) which is investigated in the simulations depends on the stage 1 sample size only via $\hat{\sigma}_{T}^{2}$. The more variation we see in $\hat{\sigma}_{T}^{2}$ the more variable second stage sample sizes will be. In turn this variability in stage 2 sample sizes induces $\alpha$ inflation.

To have a closer look, we investigate how the percentage of rejections depends on $\hat{\sigma}_{T}^{2}$ for an exemplary case with $\delta_{0}/\sigma=1$ between a fixed design and 2-stage design with SSR. Results are from $10^{5}$ simulations assuming $H_{02}:\delta=\delta_{0}$ is true. First, the responses from a total of 30 subjects, 15 per group, are generated. For the fixed design, ordinary $t$-tests are conducted from these 30 subjects. For the 2-stage design with SSR, the total variance $\hat{\sigma}_{T}^{2}$ is calculated from the 30 subjects, and a stage 2 sample size $m$, where $0\leq m<\infty$, is determined according to Equation (3) with $\alpha=5\%$ and $\beta=10\%$. Data from the two stages are then combined to conduct the $t$-test across both stages.

Figure 4 shows the percent of rejections of $H_{02}$ is related to the distribution of values of $\hat{\sigma}_{T}^{2}$ for both the fixed design (black bars) and the 2-stage design (gray bars). The height of the bars shows the percentage of rejections which occur in intervals of observed $\hat{\sigma}_{T}^{2}$ values. The first of these intervals contains all values of $\hat{\sigma}_{T}^{2}$ which are so small that $m=0$. In this example, this requires that $\hat{\sigma}_{T}^{2}\leq 0.693$, and includes 2.3% of all simulations. When this happens, the fixed and the 2-stage design are the same. $46\%$ of the $2.3\%$ where $\hat{\sigma}_{T}^{2}\leq 0.693$ (i.e. 1.06% of all cases) in the simulation are rejections of $H_{02}$. The remaining cases (where $\hat{\sigma}_{T}^{2}>0.693$ and hence $m>0$) are binned into 9 equally sized intervals, i.e. each interval contains 9,771 simulation results.

The plot supports the results from Section 2 which show that small values of $\sigma_{T}^{2}$ are providing information against $H_{02}$. As $\sigma_{T}^{2}$ gets larger, the percent rejections by the fixed design are decreasing towards 0: the weighted average across all intervals is of course $\alpha=5\%$. For the 2-stage design, the rejection rate is the same as the fixed design when $\hat{\sigma}_{T}^{2}\leq 0.693$ (because $m=0$ there), but as $\hat{\sigma}_{T}^{2}$ increases, they diverge from each other. While the second (from the left) gray bar is even lower than its black counterpart, from the 4th pair onwards the 2-stage rejection rates are higher. This reflects the fact that additional stage 2 data helps to overcome the dampening effect of large $\sigma_{T}^{2}$ on rejection probability. Overall, the weighted average of the gray bars becomes $5.83\%$ demonstrating the type I error inflation.

The graph also shows that while general tendencies are predictable—such that rejection rates must decrease; the two sets of rates diverge as we move to the right; the gray bars on average are higher than the black bars especially on the right of the plot, etc.—some peculiarities of the specific sample size formula in Equation (3) also play a role. In this case, for example, the largest total sample size requested by any of the simulation runs is 74. This shows that, although we have not put any “formal” upper limit on stage 2 sample size, the formula is such that stage 2 sample sizes which completely dominate the stage 1 data simply do not occur. If the sample size formula were such that huge stage 2 sample sizes would occur frequently enough, the red bars would approach a height of $5\%$ as we move to the right in the plot. The fact that the 2-stage design has a slightly lower rejection rate in the second lowest interval of $\sigma_{T}^{2}$ can be explained as such.

In the case when $\tilde{n}$ is large, the observed range for $\hat{\sigma}_{T}^{2}$ is much smaller. The behavior of the observed $\alpha$ values resembles a fixed design more closely. The inflation of $\alpha$ is less severe.

As this reasoning shows, the variability of $\hat{\sigma}_{T}^{2}$ in the simulations is a good indicator of the SSR method’s ability to react to different stage 1 data situations in favor of rejection of $H_{0}$. In contrast, when $\hat{\sigma}_{T}^{2}$ is very precise, the total (and the second stage) sample size is almost always the same from simulation to simulation, and we are led to always need the same overall sample size. This resembles a fixed stage design where the total sample size is predetermined and a mid-way look of the data does not change any element of the study design, hence, having no impact on $\alpha$.

Therefore, one obvious remedy in reducing $\alpha$ inflation is to increase the stage 1 sample size. However, in practice stage 1 sample size has to be reasonably small so that operationally there is enough time for an interim analysis to be carried out. Alternatively, we could reduce the variability of the stage 2 sample size by imposing a $n_{Min}$ or a $n_{Max}$ on the overall sample size. Of course, this is against the original motivation for the sample size review whose intent is to be able to modify the final actual sample size as freely as possible.

5.2 Effect size and $\alpha$ inflation

From Equations (1) and (3), with $\sigma=1$, $D=0$, and under $H_{02}:\delta=\delta_{0}$, we derive the expected value of the total sample size for each group

[TABLE]

and its standard deviation

[TABLE]

Although both $E(\hat{N})$ and $SD(\hat{N})$ approach infinity as $\delta_{0}$ goes to 0, the former approaches at a higher rate since

[TABLE]

For small $\delta_{0}$, stage 2 sample size will always overwhelm stage 1 sample size if there is no restriction on the overall sample size. In our simulations, when $\delta_{0}=0.05$ the average total sample size required to reject the null hypothesis with 90% power at 5% nominal $\alpha$ level is close to 9,000 per group. Faced with such a large average $\hat{N}$, the information carried in stage 1 data are eclipsed. This again resembles a study with a single stage.

Figure 5 shows the effect size ($x$-axis) vs. percent of Case 1, or type I error ($y$-axis) for an EQ test for all the $\tilde{n}$ we investigate, allowing $0\leq m<\infty$. At the small end of $\delta_{0}$ regardless of the size of $\tilde{n}$, $\alpha$ is around 5% for the reason just stated.

However, in reality the upper limit of the total sample size cannot be left un-capped. When $n_{Min}$ and $n_{Max}$ are predefined, since $\hat{N}$ is so large, $n_{Min}$ does not play any role, and $n_{Max}$ is always reached. What is more, since $n_{Max}<<\infty$, there is almost no power to reject $H_{01}$, which is false. This implies Case 2 is the case predominating the false rejection of $H_{02}$. For example, Figure 6 gives results for $\tilde{n}=15$ and $n_{Min}$ fixed at 30 whereas $n_{Max}$ is infinity, 60, 38, or 30. As $n_{Max}$ decreases from 60 to 38 and then 30, the power for rejecting the false $H_{01}$ decreases, and Case 1’s become Case 2’s. Visually, the effect size where Case 1 rises to the range of 5% shifts from 0.45, to 0.60 and 0.70, respectively.

As the effect size moves away from 0, $\hat{N}<n_{Max}$ more and more often, and the variability in stage 2 sample size increases. Therefore, $\alpha$ starts to become more than its nominal level for the same reason given in Section 5.1.

At the other end of the spectrum where $\delta_{0}$ is large, Equation (5) gives us

[TABLE]

and

[TABLE]

With $\beta=0.10$ and $\alpha=0.05$, when $\tilde{n}=1$, $\lim_{\delta_{0}\rightarrow\infty}\hat{N}=11$; when $\tilde{n}$ gets larger it converges to 5.5. In any practical setting where $\tilde{n}$ is at least 6, the stage 1 sample is already sufficient for the desired power. Therefore, any $n_{Min}\geq\tilde{n}\geq 6$ will have $n_{Min}$ as the final total sample size, and $n_{Max}$ will not play a role. This is why in Figure 5 where $0\leq m<\infty$, we can see a convergence toward 5% for all $\tilde{n}$ at the larger end of $\delta_{0}$.

Before $\delta_{0}$ becomes so large that $\hat{N}$ settles toward 5.5, $SD(\hat{N})$ is larger than 0, which can result in $\hat{N}$ that are above $n_{Min}$ but not equal to or exceeding $n_{Max}$. To reduce the variation in $\hat{N}$ and $\alpha$ inflation, we can set $n_{Min}$ just high enough to include all foreseeable cases where this lower limit of $\hat{N}$ suffices to reject $H_{0}$ with desired power at the nominal $\alpha$ level. From Figure 7 we can see that even with a moderate increase in the mandatory minimum total sample size, the $\alpha$ inflation reduces sharply.

5.3 The peak $\alpha$ inflation

As discussed in Section 2, the precise type I error inflation depends on many factors such as stage 1 sample size, the equivalence margin, the sample size re-estimation rule which potentially includes upper or lower limits of the final sample size. It is difficult to give a general, analytical solution to the upper limit of the $\alpha$ inflation. As can be seen in all the figures, different versions of the SSR have led to different peaks both in terms of magnitude and the corresponding $\delta_{0}$ value even with the same stage 1 sample sizes. Given the results from our simulation, if at the planning stage of a study, the range of assumed effect sizes falls on the smaller side of the effect size where peak $\alpha$ would occur, then it is important to impose a $n_{Max}$, since it can “postpone” the Case 1 from coming into the picture along the axis of effect size. It is clear though this solution results in having hardly any power for establishing the EQ or NI. Moreover, from a practical perspective, this is probably very rarely a relevant case. Because if it so happens that $\sigma^{2}$ is much larger in relation to $\delta_{0}$, it is simply hard to establish equivalence in the first place. In fact, in biosimilar pharmacokinetics studies where intra-individual variability is too large relative to the regulatory agencies’ pre-defined equivalence range, FDA for example allows an approach where the reference range are scaled to the variability of the reference product [12].

On the other hand, if the assumed effect size for the study is on the larger side of where the peak $\alpha$ would occur, it is important to impose a larger $n_{Min}$ so it can force the convergence to the nominal $\alpha$ level earlier in terms of effect size. It is obvious that this solution can result in an overpowered study that is larger than necessary.

Table 2 shows the effect sizes and the peak $\alpha$ for each $\tilde{n}$ in our simulations where $0\leq m<\infty$. They can serve as a reference for future planning. Although Table 2 refers to the type I error rate of an EQ test, for the NI test, the $\alpha$ are negligibly larger.

6 Summary

We have investigated causes of type I error inflation arising when blinded sample size reestimation is performed in equivalence trials. In Section 2, we have given an explanation for why non-trivial type I error inflation can arise in this situation. The reasoning behind this explanation allows the construction of blinded sample size estimation rules which emphasize the inflation (one such rule is investigated in the Appendix). However, from an applied perspective, it is more important to understand how this phenomenon impacts sample size re-estimation rules which are used in practice.

To investigate this, we have performed extensive simulations using SSR outlined in Section 4.1. We have set up the simulations to focus on scenarios where inflation must be suspected, but that are also practically relevant.

We find that the variation of the stage 2 sample size is a good, but not the only, indicator of (the magnitude of) type I error inflation. The stage 2 sample size is an increasing function of the total variance estimated at the interim. Hence, it is more variable when stage 1 is small and when there are no limits on the minimum and the maximum sample size allowed in stage 2. When the standardized EQ margin ($\delta_{0}/\sigma$) is very strict, stage 2 sample size is invariably estimated to be large in general and no inflation can arise. Conversely, when it is very generous, the stage 2 sample size tends to be always small and converges to a constant number as the margins tend towards infinity. Hence, there can also be no inflation. For this and practical reasons, the focus of our attention is on cases where the equivalence margin is roughly the same as the standard deviation.

We observe that if stage 2 sample size is not limited by additional restrictions, type I error inflations to $6.3\%$ can arise for stage 1 sample sizes of 10 subjects per group, a small, but not entirely unrealistic number, and persist ($5.2\%$) for as many as 80 subjects per group in stage 1.

These inflations are reduced when limits are placed on the allowed stage 2 sample sizes. In the range of practically relevant EQ margins, the lower limit ($n_{Min}$) is much more important than the upper limit ($n_{Max}$) here. The upper limit has an substantial impact on type I error probability only for values of the EQ margin that are a lot smaller than the standard deviation of the data.

Our investigation shows that the more similar to a fixed design a study is, the less $\alpha$ inflation can arise. This points to designing a study with a narrower range of $n_{Min}$ and $n_{Max}$. However, we also realize that when an interim is deemed necessary when planning a study, chances are there are uncertainties in key assumptions. Not having a lot of flexibility in choosing a second stage sample size limits the opportunities to rectify these uncertainties, hence, defeats the purpose of having an interim analysis and reduces the attractiveness of a blinded SSR. For an alternative, one can consider unblinded SSR for equivalence testing, see for example [13].

As a recommendation for clinicians and data analysts who wish to incorporate a blinded SSR for a NI or EQ study, we suggest:

• (1)

Choose a reasonably large interim sample size. From our simulations and recent experiences in planning clinical studies, $\tilde{n}$ should be at least 15.

• (2)

Allow $m\geq\tilde{n}$. In other words, impose a minimum cap $n_{Min}$ such that $n_{Min}\geq 2\tilde{n}$. If the stage 1 sample size is larger than 30, this can be relaxed.

• (3)

Limit the absolute maximum total. Our simulations show that $n_{Max}\leq 3\tilde{n}$ provides reasonable $\alpha$ control.

When all these conditions are satisfied, maximum $\alpha$ can be limited to within 5.3%. The third recommendation will probably be followed in practice already to avoid over costly big studies.

Appendix A Numerical evaluation of type I error inflation for NI and EQ testing

Here we aim to demonstrate two points. First, we give a numerical example of a sample size re-estimation rule where an increased stage 1 sample size does not imply asymptotic decreasing of $\alpha$ inflation. Second, through this example we illustrate the do-ability of analytical computation of the exact $\alpha$ once the components of an SSR rule are defined.

In this example, we assume that no additional subjects are recruited if the total variance estimate $\hat{\sigma}_{T}^{2}$, defined in Equation (1), is less than or equal to a threshold $c^{*}$. When this happens, the study is unblinded, and a final hypothesis testing is conducted based on $d=\bar{y}_{1}-\bar{y}_{2}$, the observed difference between the mean responses in group 1 and group 2 after stage 1–the only stage, of the trial.

Since $\hat{\sigma}_{T}^{2}=\frac{Q_{1}+Q_{2}}{n-1}$, we can express this as a boundary $c=(n-1)\cdot c^{*}$ on $Q_{1}+Q_{2}$.

For ease of notation, in this section, let stage 1 sample sizes for the two groups be $n_{1}$ and $n_{2}$ such that $n_{1}=n_{2}$, and total stage 1 sample size $n=n_{1}+n_{2}=2n_{1}$. Therefore, $\sqrt{\frac{n_{1}}{2}}d\sim N(\sqrt{\frac{n_{1}}{2}}\delta,\ \sigma^{2})$, and $Q_{2}=\frac{n_{1}}{2}d^{2}$. Consider the test

[TABLE]

which will be rejected if

[TABLE]

It follows that $P(\mathrm{rejecting\ }H_{02})=P\left(t_{up}\leq t_{\alpha}(n-2)\right)$ can be calculated in two steps: (1) the probability when $Q_{1}+Q_{2}\leq c$, and (2) the probability when $Q_{1}+Q_{2}>c$.

As a numerical example, consider $\alpha=0.05$, $\sigma^{2}=1$, $\delta=\delta_{up}=0.5$. We look at the cases where $n_{1}=12,\ 24,\ 40$ and let $c=n-1+\frac{n_{1}}{2}\delta^{2}$. We first calculate P\left(t_{up}\leq t_{\alpha}(n-2)\ \big{|}\ Q_{1}+Q_{2}\leq c\right).

In general, for a given $Q_{1}=x$, we have

[TABLE]

where $\Phi(\cdot)$ denotes the cdf of $N(0,1)$, and $q=t_{\alpha}(n-2)$.

Conditional on a given $x$ with $x<c$, the joint distribution of rejecting $H_{02}$ and $Q_{1}+Q_{2}\leq c$ is given by

[TABLE]

if $q\cdot\sqrt{\frac{x}{n-2}}>-\sqrt{c-x}-\sqrt{\frac{n_{1}}{2}}\delta_{up}$ and 0 otherwise.

Hence, the joint distribution of rejection of $H_{02}$ and $Q_{1}+Q_{2}\leq c$ is given by

[TABLE]

where $f(\cdot)$ denotes the pdf of $\chi^{2}(n-2)$ and $0\leq l^{*}<c^{*}\leq c$ limit the range of integration in such a way that the integrand is larger than 0. The limits $l^{*}$ and $c^{*}$ are those solutions of the two equations

[TABLE]

and

[TABLE]

which fall inside the interval $[0,c]$ (if there are such solutions). If no solution falls inside the interval $[0,c]$, then $l^{*}=0$ and $c^{*}=c$, respectively. Equations (8) and (9) can both be converted into quadratic equations and thus have closed-form solutions. To find a solution, we first have to check whether the solutions of the quadratic equations corresponding to (8) or (9) are actually solutions to (8) or (9), and then check which of these solutions fall into $[0,c]$. This is a mathematically straightforward, but somewhat tedious process.

The conditional distribution of rejection of $H_{02}$ given $Q_{1}+Q_{2}\leq c$ is thus

[TABLE]

Formula (A) can be evaluated numerically (e.g. by SAS CALL QUAD). Likewise $P\left(Q_{1}+Q_{2}\leq c\right)$ is a probability from the non-central $\chi^{2}$-distribution $\chi^{2}(n-1;\frac{n_{1}}{2}\delta^{2})$ that is implemented in most statistical software packages like SAS or R.

In the given numerical example, we have (calculations for $n_{1}=12$, numbers in brackets for $n_{1}=24,40$, respectively)

[TABLE]

Now we calculate the probability of rejecting $H_{02}$ when $Q_{1}+Q_{2}>c$. If the sample size in both groups is increased in such a way that stage 1 is rendered almost irrelevant. Then, P\left(H_{02}\mbox{ rejected}\ \big{|}\ Q_{1}+Q_{2}>c\right)\approxeq 0.05 and hence the type I error probability of this blinded SSR procedure at $\delta=\delta_{up}$ becomes $0.5946\cdot 0.0674+(1-0.5946)\cdot 0.05=0.0603(0.0612,0.0617)$ instead of the nominal $\alpha=0.05$.

Up until now, the discussion in this section focuses on NI testing when there is only one hypothesis, i.e. $H_{02}$, to be tested. Next we consider EQ testing where $H_{01}$ also need to be tested. Compare to NI testing, there is a decrease of the type I error which arises in the TOST strategy when $H_{02}$ is rejected but $H_{01}$ is not, i.e. Case 2 as defined in Section 3. This feature of TOST safe-guards the $\alpha$ inflation to some extend.

The joint probability that $Q_{1}+Q_{2}\leq c$, and both $H_{01}$ and $H_{02}$ are rejected is given by

[TABLE]

where $Z=\sqrt{\frac{n_{1}}{2}}(d-\delta_{up})\sim N(0,1)$, $q^{*}(x)=-q\sqrt{\frac{x}{n-2}}-\sqrt{\frac{n_{1}}{2}}(\delta_{up}-\delta_{low})$ and $0\leq l^{*}<c^{*}\leq c$ is such that over the range of $x\in\left[l^{*},c^{*}\right]$,

[TABLE]

For the example, let $\delta_{low}=-\delta_{up}$. Then we obtain (calculations for $n_{1}=12$, numbers in brackets for $n_{1}=24,40$, respectively):

[TABLE]

and

[TABLE]

is thus

[TABLE]

Finally, the unconditional rejection probability of both null hypotheses is

[TABLE]