Study designs for extending causal inferences from a randomized trial to a target population

TL;DR

This paper analyzes various study designs for extending causal inferences from randomized trials to broader target populations, focusing on identification methods and sampling assumptions.

Contribution

It compares nested and non-nested trial designs, detailing how sampling knowledge affects causal inference and identification strategies.

Findings

Identification depends on knowledge of sampling probabilities.

Nested and non-nested designs have different identification conditions.

Implications for estimating trial participation probabilities are discussed.

Abstract

We examine study designs for extending (generalizing or transporting) causal inferences from a randomized trial to a target population. Specifically, we consider nested trial designs, where randomized individuals are nested within a sample from the target population, and non-nested trial designs, including composite dataset designs, where a randomized trial is combined with a separately obtained sample of non-randomized individuals from the target population. We show that the causal quantities that can be identified in each study design depend on what is known about the probability of sampling non-randomized individuals. For each study design, we examine identification of potential outcome means via the g-formula and inverse probability weighting. Last, we explore the implications of the sampling properties underlying the designs for the identification and estimation of the probability of trial participation.

Tables

Table 1. Table 1: Summary of sampling properties and identification results by study design.

Study design	Sampling probabilities	Marginal probability of trial participation	Conditional probability of trial participation	Identifiable potential outcome means (when the conditions in Section 2 hold)
Nested-trial	$\mathrm{Pr}\left[D=1|S=1\right]=1$ and $\mathrm{Pr}\left[D=1|S=0\right]=1$	$\mathrm{Pr}\left[S=1\right]={\left\{1+\frac{\mathrm{Pr}\left[S=0|D=1\right]}{\mathrm{Pr}\left[S=1|D=1\right]}\right\}}^{-1}$	$\mathrm{Pr}\left[S=1|X\right]=\mathrm{Pr}\left[S=1|X,D=1\right]$	$\text{E}\left[Y^a\right]$ and $\text{E}\left[Y^a|S=0\right]$
	$\mathrm{Pr}\left[D=1|S=1\right]=1$ and $\mathrm{Pr}\left[D=1|X,A,Y,S=0\right]=\mathrm{Pr}\left[D=1|S=0\right]=c>0$, $c$ is a known constant	$\mathrm{Pr}\left[S=1\right]={\left\{1+\frac{\mathrm{Pr}\left[S=0|D=1\right]}{\mathrm{Pr}\left[S=1|D=1\right]}\times c^{-1}\right\}}^{-1}$	$\mathrm{Pr}\left[S=1|X\right]={\left\{1+\frac{\mathrm{Pr}\left[S=0|X,D=1\right]}{\mathrm{Pr}\left[S=1|X,D=1\right]}\times c^{-1}\right\}}^{-1}$	$\text{E}\left[Y^a\right]$ and $\text{E}\left[Y^a|S=0\right]$
Non-nested trial	$\mathrm{Pr}\left[D=1|S=1\right]=1$ and $\mathrm{Pr}\left[D=1|X,A,Y,S=0\right]=\mathrm{Pr}\left[D=1|S=0\right]=u>0,$ $u$ is an unknown constant	Not identifiable	Not identifiable	$\text{E}\left[Y^a|S=0\right]$

Equations

$$\mathrm{Pr}[D=1|S=1]=\mathrm{Pr}[D=1|S=0]=1,$$

$$\begin{array}{rl}& \mathrm{Pr}[D=1|S=1]=1\text{mbox}and\\ & \mathrm{Pr}[D=1|X,A,Y,S=0]=\mathrm{Pr}[D=1|S=0]=c>0,\end{array}$$

$$\begin{split}&\Pr[D=1|S=1]=1\mbox{ and }\\ &\Pr[D=1|X,A,Y,S=0]=\Pr[D=1|S=0]=u>0,\mbox{ where $u$ is an unknown constant.}\end{split}$$

$$\begin{array}{rl}& (X,A,Y,S=1,D=1)\text{mbox}fortrialparticipants;\\ & (X,S=0,D=1)\text{mbox}for\text{sampled}non-randomizedindividuals;\\ & (S=0,D=0)\text{mbox}for\text{non-sampled}non-randomizedindividuals.\end{array}$$

$$\begin{array}{rl}& (X,A,Y,S=1,D=1)\text{mbox}fortrialparticipants;and\\ & (X,S=0,D=1)\text{mbox}for\text{sampled}non-randomizedindividuals.\end{array}$$

$$\begin{split}\operatorname{\mbox{E}}[Y^{a}]&=\operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]}\\ &=\int\operatorname{\mbox{E}}[Y|X=x,S=1,A=a]dF_{X}(x),\end{split}$$

$$\begin{split}\operatorname{\mbox{E}}[Y^{a}|S=0]&=\operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]|S=0\big{]}\\ &=\int\operatorname{\mbox{E}}[Y|X=x,S=1,A=a]dF_{X|S}(x|S=0),\end{split}$$

$$F_X(x)=\sum _{s=0}^1{F}_{X|S}(x|S=s)\mathrm{Pr}[S=s].$$

$$\mathrm{Pr}[S=1]={\left\{1+\frac{\mathrm{Pr}[S=0|D=1]}{\mathrm{Pr}[S=1|D=1]}\times c^{-1}\right\}}^{-1}.$$

$$\mathrm{Pr}[S=1]={\left\{1+\frac{\mathrm{Pr}[S=0|D=1]}{\mathrm{Pr}[S=1|D=1]}\times u^{-1}\right\}}^{-1}.$$

$$\begin{split}\operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]}&=\operatorname{\mbox{E}}\!\left[\dfrac{I(S=1,A=a)Y}{\Pr[S=1|X]\Pr[A=a|X,S=1]}\right],\end{split}$$

$$\begin{split}\operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{|}S=0\big{]}&=\dfrac{\quad\operatorname{\mbox{E}}\!\left[\dfrac{I(S=1,A=a)Y\Pr[S=0|X]}{\Pr[S=1|X]\Pr[A=a|X,S=1]}\right]\quad}{\operatorname{\mbox{E}}\!\left[\dfrac{I(S=1,A=a)\Pr[S=0|X]}{\Pr[S=1|X]\Pr[A=a|X,S=1]}\right]}.\end{split}$$

$$\begin{array}{r}\mathrm{Pr}[S=1|X]={\left\{1+\frac{\mathrm{Pr}[S=0|X,D=1]}{\mathrm{Pr}[S=1|X,D=1]}\times c^{-1}\right\}}^{-1},\end{array}$$

$$\frac{\mathrm{Pr}[S=1|X]}{\mathrm{Pr}[S=0|X]}=\frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}\times c.$$

$$\mathrm{Pr}[S=1|X]={\left\{1+\frac{\mathrm{Pr}[S=0|X,D=1]}{\mathrm{Pr}[S=1|X,S=1]}\times u^{-1}\right\}}^{-1}.$$

$$\frac{\mathrm{Pr}[S=1|X]}{\mathrm{Pr}[S=0|X]}=\frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}\times u.$$

$$\begin{array}{rl}\mathcal{L}(\gamma )& =\prod _{i=1}^n{\left[p(X_i;\gamma )\right]}^{S_i}{\left[1-p(X_i;\gamma )\right]}^{1-S_i},\end{array}$$

$$\ell_{0}(\gamma)=\operatorname{\mbox{E}}\!\Big{[}S\log\big{[}p(X;\gamma)\big{]}+(1-S)\log\big{[}1-p(X;\gamma)\big{]}\Big{]}.$$

$$\begin{split}\mathscr{L}_{\scalebox{0.5}{\text{W}}}(\gamma)&=\prod\limits_{i=1}^{n}\big{[}p(X_{i};\gamma)\big{]}^{S_{i}D_{i}}\big{[}1-p(X_{i};\gamma)\big{]}^{[(1-S_{i})D_{i}]/c},\end{split}$$

$$\ell_{\scalebox{0.5}{\text{W}}0}(\gamma)=\operatorname{\mbox{E}}\!\left[SD\log\big{[}p(X;\gamma)\big{]}+\dfrac{[(1-S)D}{c}\log\big{[}1-p(X;\gamma)\big{]}\right],$$

$$\ln \frac{\mathrm{Pr}[S=1|X]}{\mathrm{Pr}[S=0|X]}={\beta }_0+\sum _{j=1}^p{\beta }_j{X}_j.$$

$$\begin{array}{rl}\ln \frac{\mathrm{Pr}[S=1|X]}{\mathrm{Pr}[S=0|X]}& =\ln (u)+\ln \frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}.\end{array}$$

$$\begin{array}{rl}\ln \frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}& ={\beta }_0^{\ast }+\sum _{j=1}^p{\beta }_j{X}_j,\end{array}$$

$$\mathrm{Pr}[S=1]=\frac{\mathrm{Pr}[S=1|D=1]\mathrm{Pr}[D=1]}{\mathrm{Pr}[D=1|S=1]}\text{mbox}and\mathrm{Pr}[S=0]=\frac{\mathrm{Pr}[S=0|D=1]\mathrm{Pr}[D=1]}{\mathrm{Pr}[D=1|S=0]}.$$

$$\begin{array}{rl}\frac{\mathrm{Pr}[S=1]}{\mathrm{Pr}[S=0]}& =\frac{\mathrm{Pr}[S=1|D=1]}{\mathrm{Pr}[S=0|D=1]}\times \frac{\mathrm{Pr}[D=1|S=0]}{\mathrm{Pr}[D=1|S=1]}\\ & =\frac{\mathrm{Pr}[S=1|D=1]}{\mathrm{Pr}[S=0|D=1]}\times c.\end{array}$$

$$\mathrm{Pr}[S=1]={\left\{1+\frac{\mathrm{Pr}[S=0|D=1]}{\mathrm{Pr}[S=1|D=1]}\times c^{-1}\right\}}^{-1}.$$

$$\begin{array}{rl}\mathrm{Pr}[S=1|X]& =\frac{\mathrm{Pr}[S=1|X,D=1]\mathrm{Pr}[D=1|X]}{\mathrm{Pr}[D=1|X,S=1]}\text{mbox}and\\ \mathrm{Pr}[S=0|X]& =\frac{\mathrm{Pr}[S=0|X,D=1]\mathrm{Pr}[D=1|X]}{\mathrm{Pr}[D=1|X,S=0]}.\end{array}$$

$$\begin{array}{rl}\frac{\mathrm{Pr}[S=1|X]}{\mathrm{Pr}[S=0|X]}& =\frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}\times \frac{\mathrm{Pr}[D=1|X,S=0]}{\mathrm{Pr}[D=1|X,S=1]}\\ & =\frac{\mathrm{Pr}[S=1|X,D=1]}{\mathrm{Pr}[S=0|X,D=1]}\times c.\end{array}$$

$$\begin{array}{r}\mathrm{Pr}[S=1|X]={\left\{1+\frac{\mathrm{Pr}[S=0|X,D=1]}{\mathrm{Pr}[S=1|X,D=1]}\times c^{-1}\right\}}^{-1}.\end{array}$$

$$\begin{split}\mathscr{L}(\gamma)&=\prod\limits_{i=1}^{n}\big{[}p(X_{i};\gamma)\big{]}^{S_{i}}\big{[}1-p(X_{i};\gamma)\big{]}^{1-S_{i}},\end{split}$$

Full text

A number of recent contributions [1, 2, 3, 4, 5, 6, 7] have discussed methods for addressing problems related to selective study participation [8] in randomized trials. These methods can be used to extend (i.e., generalize or transport [9]) causal inferences from a randomized trial to a target population. The methods require baseline covariate, treatment, and outcome data from individuals participating in the trial and baseline covariate data from non-randomized individuals. Estimation of potential outcome means in the target population typically requires models for the probability of trial participation [1], the expectation of the outcome under each treatment among trial participants [5], or both (to improve robustness [3, 7]). Prior work has largely focused on identifiability conditions and estimation approaches, without a clear connection to study design principles, obscuring the fact that different study designs determine which causal quantities can be identified and have implications for identifying and estimating the conditional probability of trial participation.

Two types of study designs that combine data from randomized individuals with data from a sample of non-randomized individuals have been used for the explicit goal of estimating potential outcome means and treatment effects in a well-defined target population: (1) nested trial designs, in which the randomized trial is embedded in a sample from the target population [3]; and (2) non-nested trial designs, in which observations from randomized individuals are combined with a separately obtained sample of non-randomized individuals from the target population. The sampling probability of non-randomized individuals is known in nested trial designs [2], but unknown in non-nested trial designs [4, 6, 7]. In both types of study designs, baseline covariate data are collected from all randomized individuals and from sampled non-randomized individuals; treatment and outcome data need only be collected from randomized individuals. Though treatment and outcome data from non-randomized individuals can be used to evaluate assumptions or improve efficiency, they are not necessary for identification and estimation under the assumptions used in this paper or the bulk of the related literature (e.g., as reviewed in [5]).

In this paper, we show that the different causal quantities that can be identified in each study design depend on what is known about the probability of sampling non-randomized individuals. For each study design, we examine identification of potential outcome means via the g-formula and inverse probability weighting, and explore the implications of the design’s sampling properties for modeling the probability of trial participation.

1 Sampling properties and the observed data

For a well-defined causal question, investigators can specify a set of eligibility criteria that define an actual population of individuals to whom research findings would be applicable, in the sense that we can in principle identify all individuals who meet the criteria. For instance, when designing a randomized trial, the trial eligibility criteria define an actual population of all trial-eligible individuals. In this paper, we view the actual population as a simple random sample from an (infinite) superpopulation of individuals; we refer to this superpopulation as the target population [10]. We are interested in causal quantities that pertain to the target population or to its subsets (e.g., defined by trial participation status).

To introduce some notation, let $X=(X_{1},\ldots,X_{p})$ denote a vector of $p$ baseline covariates; $A$ the treatment assignment indicator; $Y$ the observed outcome; and $S$ the trial participation indicator, with $S=1$ for randomized individuals and $S=0$ for non-randomized individuals (individuals who are either not invited to participate in the trial or who are invited but decline). To capture the notion that some non-randomized individuals in the actual population ($S=0$) may not be sampled, let $D$ be an indicator for whether an individual in the actual population is sampled and contributes data to the analyses, with $D=1$ for sampled individuals and $D=0$ for non-sampled individuals..

We can now describe the sampling properties that underlie nested and non-nested study designs. These properties describe how the observed study sample relates to the actual population; the underlying actual population and (hypothetical) target population are the same. Figure 1 illustrates the conceptual relationships between designs, their sampling properties, and the observed data.

In the main text of this paper, we consider simple random samples, with known or unknown sampling probabilities, from the actual population or from the non-randomized subset of the actual population. As we discuss below, our main results, with minor modifications, hold when the sampling probability is a known function of auxiliary baseline covariates rather than a known constant (i.e., when we have random sampling, not simple random sampling). Allowing the sampling probabilities to depend on auxiliary covariates, however, does not lead to additional insights regarding study design [11]; for this reason, in the main text, we assume that the sampling probability does not depend on covariates.

1.1 Nested trial designs

We consider two variants of the nested trials design: when a census of the actual population is taken and when the non-randomized individuals are sub-sampled.

Census of the actual population: In this variant of the nested trial design, the individuals contributing data to the analysis are assumed to be a census of the actual population, that is,

[TABLE]

thus, nested trial designs can be viewed as simple random samples from the superpopulation. In this design, it is common to define the target population implicitly, based on the actual population in which the trial is nested. For example, in comprehensive cohort studies [12], investigators nest a trial within a cohort of all individuals who met the trial’s eligibility criteria and were invited to participate in the trial. They then define the target population as the population from which cohort members (i.e., the actual population of trial-eligible individuals invited to participate in the trials) could have been a simple random sample. Thus, in this design, investigators need to ensure that the cohort represents the target population they are interested in; that is, the trial eligibility criteria need to be broad enough to address the research question and the individuals invited to participate in the trial (who form the cohort in which the trial is nested) need to represent the target population of interest.

Sub-sampling of non-randomized individuals: In this variant, we collect data from all randomized individuals in the actual population but only collect baseline covariate data from a sub-sample of the non-randomized individuals in the actual population, with sampling probability that is a known constant. The sampling properties can be summarized as

[TABLE]

where $c$ is a known constant, with $0<c\leq 1$. Note that the nested trial design with a census of the actual population can be viewed as a special case of the sub-sampling design, with $c=1$. Using $c<1$ is statistically less efficient than using $c=1$, but may improve research economy, for example, if the collection of covariate data among non-randomized individuals is expensive [11]. Furthermore, as noted, a variant of the nested trial design with sub-sampling allows the selection of non-randomized individuals to depend on auxiliary baseline covariates; we show how our results extend to that case in the Appendix.

1.2 Non-nested trial designs

In non-nested trial designs, data from randomized and non-randomized individuals are obtained separately. Investigators assume that data from all randomized individuals can be combined with data from a simple random sample of non-randomized individuals from the actual population, with sampling probability that was an unknown constant (e.g., [4]). The sampling properties can be summarized as

[TABLE]

An example of non-nested trial design is the composite dataset design [4, 7]. Here, investigators append the data from a randomized trial to data from a convenience sample of non-randomized individuals, often obtained from routinely collected data sources (such as claims or electronic medical records databases, or prospective cohort studies). The assumption, often left unstated, is that the sample of non-randomized individuals is a simple random sample from the population of non-randomized individuals (or a well-defined subset thereof) to whom the investigators wish to extend the trial results [7, 4].

1.3 The observed data

In both nested and non-nested designs, we collect data on baseline covariates, treatment, and outcome from randomized individuals; in contrast, as we show in Section 3, only baseline covariate data are needed from non-randomized individuals.

More specifically, for nested designs the observed data consist of realizations of

[TABLE]

Because all randomized individuals are sampled, we have that $(D=1,S=1)\Leftrightarrow(S=1)$. No covariate, treatment, or outcome data are available for non-sampled non-randomized individuals, $(D=0,S=0)$. Note also that in nested trial designs with a census of the actual population, the $(D=0,S=0)$ subset does not exist.

In non-nested trial designs (e.g., composite dataset designs), we typically do not know the number of non-sampled non-randomized individuals, thus the observed data consist of realizations only of

[TABLE]

2 Causal quantities of interest and identifiability conditions

2.1 Causal quantities of interest

In order to define causal quantities, let $Y^{a}$ be the potential (counterfactual) outcome under intervention to set treatment to $a$ [13, 14]. We are interested in the mean of these potential outcomes in the target population $\operatorname{\mbox{E}}[Y^{a}]$ or in the non-randomized subset of the target population $\operatorname{\mbox{E}}[Y^{a}|S=0]$. For example, $\operatorname{\mbox{E}}[Y^{a}]$ captures the outcome under the strategy of treating all individuals in the target population with $a$. And it is often scientifically and methodologically interesting to compare $\operatorname{\mbox{E}}[Y^{a}|S=0]$ against $\operatorname{\mbox{E}}[Y^{a}|S=1]$, to examine whether the potential outcome mean under treatment $a$ differs among trial-participants and non-participants in the target population [3].

2.2 Identifiability conditions

For all study designs, the following identifiability conditions are sufficient to extend inferences from a clinical trial to a target population [3, 7]:

(1) Consistency of potential outcomes: interventions are well-defined, so that if $A_{i}=a,$ then $Y^{a}_{i}=Y_{i}$. Implicit in this notation is that the offer to participate in the trial and trial participation itself do not have an effect on the outcome except through treatment assignment (e.g., there are no Hawthorne effects).

(2) Mean exchangeability among trial participants: $\operatorname{\mbox{E}}[Y^{a}|X,S=1,A=a]=\operatorname{\mbox{E}}[Y^{a}|X,S=1]$. This condition is expected to hold because of randomization (marginal or conditional on $X$).

(3) Positivity of treatment assignment in the trial: $\Pr[A=a|X=x,S=1]>0$ for each $a$ and each $x$ with positive density among randomized individuals, $f_{X|S}(x|S=1)>0$. This condition is also expected to hold because of randomization.

(4) Mean generalizability (exchangeability over $S$): $\operatorname{\mbox{E}}[Y^{a}|X,S=1]=\operatorname{\mbox{E}}[Y^{a}|X]$ for each $a$. For binary $S$, this condition implies the mean transportability condition $\operatorname{\mbox{E}}[Y^{a}|X,S=1]=\operatorname{\mbox{E}}[Y^{a}|X,S=0]$ (provided both conditional expectations are well-defined).

(5) Positivity of trial participation: $\Pr[S=1|X=x]>0$ for each $x$ with positive density in the target population, $f_{X}(x)>0$.

In the conditions listed above, we have used $X$ generically to denote baseline covariates. It is possible, however, that strict subsets of $X$ are adequate to satisfy different exchangeability conditions. For example, in a marginally randomized trial the mean exchangeability among trial participants holds unconditionally. Furthermore, to focus on issues related to selective trial participation, we will assume complete adherence to the assigned treatment and no loss-to-follow-up.

2.3 Trial eligibility criteria and choice of target population

Now that we have specified the causal quantities of interest and listed identifiability conditions, we can consider the choice of target population in more detail. As noted, the target population should be determined by the scientific question investigators hope to address. In many cases, when using the methods described in this paper, it is sensible to limit the target population to the population of individuals meeting the trial eligibility criteria or to a subset of that population. To the extent that the variables used to define the trial eligibility criteria are needed to satisfy the mean generalizability condition, the restriction to trial-eligible individuals is needed for the positivity of trial participation condition to hold – individuals not meeting the criteria are not allowed to enter the trial. In some cases, however, investigators may be able to argue that only a subset of the variables used to determine trial eligibility are necessary for the mean generalizability condition to hold. In such cases, the target population can be broader than the population of trial eligible individuals. The essential requirement is that the distributions of covariates needed to satisfy the mean generalizability condition among randomized and non-randomized individuals should have common support.

3 Identification via the g-formula

We begin by considering identification by the g-formula [15]. Using the identifiability conditions of Section 2, it is straightforward to show that the potential outcome mean in the target population [3] can be re-expressed as

[TABLE]

where $F_{X}(x)$ denotes the distribution of $X$ in the target population.

The potential outcome mean among non-randomized individuals in the target population [7] can be re-expressed as

[TABLE]

where $F_{X|S}(x|S=0)$ denotes the distribution of $X$ among non-randomized individuals in the target population (i.e., the subset with $S=0$).

First, we note that both results involve the conditional expectation of the outcome $Y$ among trial participants assigned to treatment $a$, $\operatorname{\mbox{E}}[Y|X,A=a,S=1]$. Because both nested and non-nested designs assume that all randomized individuals are sampled, this expectation is identifiable in both designs.

Next, we turn out attention to the identification of $F_{X}(x)$ and $F_{X|S}(x|S=0)$, which are necessary to identify $\operatorname{\mbox{E}}[Y^{a}]$ and $\operatorname{\mbox{E}}[Y^{a}|S=0]$, respectively. There are interesting differences between the designs when it comes to identifying these distributions and we consider each design individually below.

3.1 Nested trial designs

Census of the actual population: Identification is most straightforward in this design, because data are available from all members of the actual population (both randomized and non-randomized) and the actual population is a simple random sample from the target population. Thus, $F_{X}(x)$ is identifiable. Furthermore, in this design, every subgroup of the actual population defined on the basis of baseline covariates or trial participation is a simple random sample from the corresponding subgroup in the target population. Thus, the distribution of covariates among non-randomized individuals $F_{X|S}(x|S=0)$ can also be identified. It follows that all the components on the right-hand-sides of (1) and (2) are identifiable, establishing that \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]} and \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{|}S=0\big{]} are identifiable.

Sub-sampling of non-randomized individuals: For this design, identification of the marginal distribution of $X$ is slightly more involved because the non-randomized individuals contributing data to the analysis are a sub-sample from the non-randomized individuals in the actual population.

First, by the law of total probability, for binary $S$,

[TABLE]

Clearly, $F_{X|S}(x|S=s)$, for $s=0,1$ is identifiable because the randomized and non-randomized sampled individuals are simple random samples of the target population subsets with $S=1$ and $S=0$, respectively. The only difficulty, then, is identification of the marginal probability of trial participation, $\Pr[S=1]$, because $\Pr[S=0]=1-\Pr[S=1]$. As we show in the Appendix, under the sampling properties of the nested trial design with sub-sampling of non-randomized individuals,

[TABLE]

The odds of non-participation in the trial among sampled individuals, $\dfrac{\Pr[S=0|D=1]}{\Pr[S=1|D=1]}$, are identifiable; and, as defined in Section 1.1, $c$ is a known constant. It follows that $F_{X}(x)$ is identifiable and, consequently, \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]} is also identifiable because all the components of the integral on the right-hand-side of (1) are identifiable.

Turning our attention to $F_{X|S}(x|S=0)$, we note that it is identifiable because the sampled non-randomized individuals are a simple random sample from the non-randomized individuals in the actual population. It follows that \operatorname{\mbox{E}}\!\big{[}\!\operatorname{\mbox{E}}[Y|X,S=1,A=a]|S=0\big{]} is identifiable because all the components of the integral on the right-hand-side of (2) are identifiable.

3.2 Non-nested trial designs

Using an argument parallel to that for nested trial designs with sub-sampling, when the probability of sampling a non-randomized individual is unknown, the probability of trial participation, $\Pr[S=1]$, can be expressed in the form of equation (3), substituting the $u$ for $c$,

[TABLE]

Because, as defined in Section 1.2, $u$ is an unknown constant, $F_{X}(x)$ is not identifiable and consequently \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]} is also not identifiable.

Turning our attention to $F_{X|S}(x|S=0)$, we see that it is identifiable because the non-randomized individuals contributing data to the analysis are a simple random sample from the non-randomized individuals in the actual population (even though the sampling probability is unknown). It follows that \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{|}S=0\big{]} is identifiable in non-nested trial designs because all the components of the integral in (2) are identifiable.

4 Identification via IP weighting

There has been considerable recent interest [1, 3, 2, 4, 7] in using weighting methods to identify the potential outcome means in equations (1) and (2), because the specification of models for the probability of trial participation is often considered a somewhat easier task than the specification of models for the outcome among trial participants.

First, consider \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]}, which we argued is identifiable in nested trials. As shown in [3], we can re-express the right-hand-side of (1) as

[TABLE]

where $I(\cdot)$ denotes the indicator function.

Now, consider \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]|S=0\big{]}, which we argued is identifiable by the g-formula in both nested and non-nested trials. As shown in [7], we can re-express the right-hand-side of (2) as

[TABLE]

The probability of treatment among trial participants, $\Pr[A=a|X,S=1]$ is under the control of the investigators and does not pose any difficulties for identification of either functional. Now, for each design, we focus our attention on the conditional probability or the conditional odds of trial participation, which appear in expressions (4) and (5), respectively.

4.1 Nested trial designs

Census of the actual population: Identification of $\Pr[S=1|X]$ in this design is an obvious consequence of the fact that individuals contributing data to the analysis are a simple random sample from the target population. In other words, because we have sampled all individuals in the actual population, which is a simple random sample of the target population, $\Pr[S=1|X]=\Pr[S=1|X,D=1]$.

Sub-sampling of non-randomized individuals: Identification of $\Pr[S=1|X]$ is only a little more difficult when we sample non-randomized individuals from the actual population. As we show in the Appendix, under the sampling properties of this design,

[TABLE]

where the conditional odds of trial participation among sampled individuals, $\dfrac{\Pr[S=0|X,D=1]}{\Pr[S=1|X,D=1]}$, are identifiable and $c$ is a known constant defined in Section 1.1. It follows that $\Pr[S=1|X]$ is identifiable and the odds of trial participation can be written as

[TABLE]

In sum, the IP weighting re-expressions of the functionals of interest are identifiable in nested trial designs.

4.2 Non-nested trial designs

We can use an argument parallel to that for nested trial designs with sub-sampling, to establish that, when the sampling probability for non-randomized individuals is unknown, the probability of trial participation, $\Pr[S=1|X]$, can be expressed as,

[TABLE]

Because, as defined in Section 1.2, $u$ is unknown, the conditional probability of trial participation, which appears on the right hand side of (4), is not identifiable; this confirms our earlier result that \operatorname{\mbox{E}}\!\big{[}\operatorname{\mbox{E}}[Y|X,S=1,A=a]\big{]} cannot be identified in non-nested trials.

Furthermore, the conditional odds of trial participation are also not identifiable because they depend on $u$. In fact, using equation (7), substituting $u$ for $c$, we see that the odds of trial participation in the target population are, up to an unknown multiplicative constant, equal to the odds of trial participation among sampled individuals,

[TABLE]

We have come to an apparent conflict: the right hand-side of (5) involves the conditional odds of trial participation, a quantity that is not identifiable in non-nested designs. Yet, we argued in the previous section that the left-hand-side of (5) is identifiable. The conflict can be easily resolved by noting that, because both the numerator and the denominator of (5) are multiplied by the unknown constant $u$, which cancels out, identification via IP weighting is possible (see the appendix of [7] for technical details).

Table 1 summarizes the sampling properties and identification results for each study design.

5 Estimating the probability of trial participation

In realistic analyses, the dimension of $X$ will be fairly large, necessitating some modeling assumptions about $\Pr[S=1|X]$ or $\Pr[S=1|X,D=1]$ [16]. In this section we discuss the relationship between study design and model specification and estimation approaches.

5.1 Nested trial designs

Census of the actual population: In this type of nested trial design, it is straightforward to estimate the probability of trial participation, $\Pr[S=1|X]$, in the sense that we can use the model we believe is most likely to be correctly specified for the target population.

For concreteness, suppose that we are willing to assume a parametric model, $p(X;\gamma),$for the probability of trial participation in the target population, $\Pr[S=1|X]$, with $\gamma$ a finite dimensional parameter. In the nested-trial designs with a census of non-randomized individuals, we typically estimate the parameters by maximizing the likelihood function

[TABLE]

where $i=1,\ldots,n$, and $n$ is the number of individuals in the study (i.e., the actual population). Under reasonable technical conditions [17], the usual maximum likelihood methods use a sample-size normalized objective function that converges uniformly in probability to

[TABLE]

For example, when $p(X;\gamma)$ is a logistic model, $\ell_{0}(\gamma)$ is the large sample limit of the sample-size normalized log-likelihood function for logistic regression.

Sub-sampling of non-randomized individuals: When we sub-sample of the non-randomized individuals in the actual population, it is not possible to maximize the likelihood function above, because data are not available from all non-randomized individuals in the actual population. A natural idea is to use equation (6), which provides an explicit formula for identifying the conditional probability of trial participation, $\Pr[S=1|X]$ using the probability of trial participation among sampled individuals, $\Pr[S=1|X,D=1]$, and the sampling probability for non-randomized individuals, $\Pr[D=1|S=0]$. When modeling the probability of trial participation among sampled individuals, however, the following difficulty arises: in general, when sampling depends on trial participation status, the correctly specified model for trial participation does not have the same form as the correctly specified model in the target population, with the notable exception of the logistic regression model [18, 19]. This implies that naive estimation of the parameters of the model for trial participation among sampled individuals will typically be inconsistent for the population model.

Because the sampling probability of non-randomized individuals is known, we can use the following weighted pseudo-likelihood function, which only uses data from sampled individuals [18, 20],

[TABLE]

with $c=\Pr[D=1|S=0]$. Weighted maximum likelihood methods use a sample-size normalized objective function that converges uniformly in probability to

[TABLE]

which is restricted to sampled individuals ($D=1$).

As we show in the Appendix, under the sampling properties for this design, the large sample limits of the objective functions in (10) and (11) are equal, $\ell_{0}(\gamma)=\ell_{\scalebox{0.5}{\text{W}}0}(\gamma)$. It follows that, under reasonable technical conditions [17], weighted likelihood estimation of $\gamma$ in the nested trial design with sub-sampling of non-randomized individuals converges in probability to the same parameter as unweighted regression in the actual population.

In practical terms, as long as a reasonable parametric model can be specified for the target population, the model parameters can be estimated using weighted maximum likelihood methods [18] on data from sampled individuals, with individual level weights equal to 1 for randomized individuals, $S=1,D=1$; $c^{-1}$ for sampled non-randomized individuals, $S=0,D=1$; and 0 for unsampled individuals, $D=0$.

5.2 Non-nested trial designs

In non-nested trial designs, the weighting approach described above is not applicable because the sampling probability of non-randomized individuals is unknown. Provided, however, that the sampling probability does not depend on $X$ (i.e., the assumed sampling property), we can show that, if a logistic model for trial participation is correctly specified in the target population, then a logistic model is correctly specified in the non-nested trial design. To see this, suppose that we are willing to assume a logistic regression model in the population, such that

[TABLE]

Using the result in (8) and taking logarithms, we have that

[TABLE]

Equating the right-hand-sides of the last two equations, we obtain

[TABLE]

where $\beta_{0}^{*}=\beta_{0}-\ln(u)$, a well-known result in the context of case-referent studies [21]. Thus, if a logistic model is correctly specified in the target population, then a model of the same functional form is correctly specified in the non-nested trial design. In fact, the coefficients in the two models are equal, and only the intercept differs. Because $0<u<1$, $\beta_{0}^{*}>\beta_{0}$: the sub-sampling of non-randomized patients simply results in an intercept that is “shifted” upwards. As we have shown in the section on IP weighting, the resulting shift in the odds of participation does not affect identification of the potential outcome mean in the non-randomized individuals, $\operatorname{\mbox{E}}[Y^{a}|S=0]$, which is the parameter of interest in non-nested trial designs with unknown sampling probability of non-randomized individuals.

The above result is also important for estimation of the model parameters: combined with the results in [22, 23], it implies that the unconstrained and unweighted maximum likelihood estimator for the logistic model in (12), fit among sampled individuals, is the efficient estimator for $\beta_{j}$, $j=1,\ldots,p$.

6 Discussion

We presented a unified description of study designs for extending inferences from randomized trials to a well-defined target population and showed that commonly invoked identifiability conditions need to be combined with the sampling properties of each study design in order to determine which causal quantities can be identified. Our approach uses a superpopulation framework, which is a natural choice for extending trial findings beyond the sample of randomized individuals [24].

In non-nested trial designs, where the sampling probability for non-randomized individuals is unknown, the marginal potential outcome means in the target population are not identifiable, but the potential outcome means in the sub-population of non-randomized individuals are identifiable. This restriction may be less severe than it appears: for most trials, we want to estimate the effect of applying the interventions to a new population, which can be represented by a well-chosen sample of non-randomized individuals [7]. In any case, when available, knowledge of the sampling probability of non-randomized individuals can be used to mitigate these limitations, without requiring the collection of covariate information from all non-randomized individuals in the actual population. Thus, in general, nested trial designs will often be the preferred approach for generalizing trial findings when it is possible to define and sample the actual population when a randomized trial is planned. Such nested trial designs will typically have broad (pragmatic [25]) eligibility criteria and define the target population as the population of individuals meeting the trial eligibility criteria. When that is not possible, as is the case in already completed randomized trials, non-nested trial designs might be a reasonable alternative. For example, in non-nested trial designs, the comparison of estimates for the potential outcome means among randomized, $\widehat{\operatorname{\mbox{E}}}[Y^{a}|S=1]$, and non-randomized individuals, $\widehat{\operatorname{\mbox{E}}}[Y^{a}|S=0]$, is of practical interest: provided the identifiability conditions hold, if $\widehat{\operatorname{\mbox{E}}}[Y^{a}|S=1]\approx\widehat{\operatorname{\mbox{E}}}[Y^{a}|S=0]$, we may conclude that the trial results are likely generalizable (up to sampling variability); in contrast, if the estimates are different, trial results may not be generalizable.

We also showed that the different study designs have implications for identifying and estimating the conditional probability of trial participation. This probability is of inherent interest because it captures aspects of decision-making related to trial participation [26, 27]. We showed that the probability is identifiable in nested trial designs, but not in non-nested trial designs (e.g., composite dataset designs). Indeed, any reasonable parametric model for the probability of participation in the population can be identified in nested trial designs. In nested trial designs with sampling of non-randomized individuals, estimation of model parameters can be facilitated by the use of weighted maximum likelihood estimation where randomized patients are given weight 1 and non-randomized patients are given weight equal to the inverse of the probability of being sampled among non-randomized individuals in the actual population. In non-nested trial designs, model specification is complicated by the fact that, when sampling depends on trial participation status, the model for the probability of trial participation among sampled individuals is not of the same form as the model in the population (the logistic regression model being a notable exception [18]).

The probability of trial participation in the target population is also important for identification and estimation using inverse probability (or odds) weighting methods. Our argument about the odds of participation after selection of non-randomized individuals being equal to the odds of participation in the target population up to an unknown multiplicative constant, clarifies how the validity of estimators when using composite datasets designs [4, 7] depends critically on the assumed sampling properties.

Astute readers will have noticed the many connections between our results and the theory of case-referent (case-control) studies [28, 21, 18, 20, 19]. Indeed, our approach can be placed in the case-base paradigm, viewing randomized individuals as “cases” in cumulative incidence case-referent study [28] nested in the “cohort” of the actual population. An interesting parallel with case-referent studies: the difficulty in specifying the population of non-randomized individuals that should be sampled in composite dataset designs is similar in nature to the validity issues of case-referent studies with a secondary base [29, 30, 31].

In this paper, for simplicity, we focused on causal quantities that are most meaningful for point treatments with complete adherence and no loss to follow-up. Our results can be extended to address time-varying treatments using well-known extensions of the identifiability conditions for randomized trials [15, 32, 24], without any changes to the sampling properties or the modeling assumptions about the probability of trial participation. Perhaps, then, the most consequential causal assumption that we required was that the invitation to participate in the trial and trial participation itself do not have an effect on the outcome except through treatment assignment. Unless investigators are willing to contemplate much more complex study designs involving multistage data collection about (and possibly randomization of) the invitation to participate, trial participation itself, and treatment assignment [33], our results are best viewed as applying to trials where the not-through-treatment effects of the invitation to participate in the trial and of trial participation are negligible compared to the effects of treatment. For example, they are applicable to pragmatic randomized trials embedded in large health-care systems or registries, where trial procedures other than treatment assignment can be assumed to be similar to usual medical practice [34, 25, 35].

7 Figure

8 Table

Appendix A Identification of the probability of trial participation in nested trial designs with sub-sampling

A.1 Identification of the marginal probability of trial participation

Using the definition of conditional probability and re-arranging,

[TABLE]

Taking the ratio of the above expressions and exploiting the sampling properties for non-nested trial designs,

[TABLE]

With a bit of algebra, the above expression can be re-arranged to show that

[TABLE]

By setting $c=1$ we see that in the nested-trial design with a census of non-randomized individuals $\Pr[S=1]=\Pr[S=1|D=1]$.

A.2 Identification of the conditional probability of trial participation

The argument for the conditional probability is parallel to the one presented above for the marginal probability. Again, using the definition of conditional probability,

[TABLE]

Taking the ratio of the above expressions and exploiting the sampling properties for non-nested trial designs,

[TABLE]

The above expression can be re-arranged to show that

[TABLE]

By setting $c=1$ we see that in the nested-trial design with a census of non-randomized individuals $\Pr[S=1|X]=\Pr[S=1|X,D=1]$.

Appendix B Estimating the conditional probability of trial participation

We outline the proof for the convergence in probability of the estimators for the conditional probability of trial participation described in the main text, without delving into the technical conditions needed to make the arguments rigorous.

Consider the likelihood function for the nested trial design with a census of the actual population,

[TABLE]

and, the pseudo-likelihood function for the nested trial design with known sampling probability of the non-randomized individuals,

[TABLE]

For $\mathscr{L}(\gamma)$, the sample size-normalized objective function to be maximized is

[TABLE]

Provided the technical conditions for the uniform law of large numbers obtain, the above objective function converges uniformly in probability, in the sense of the definition in Section 2.1 of [17], to

[TABLE]

By Theorem 2.1 of [17], if $\ell_{0}(\gamma)$ is uniquely maximized at $\gamma_{0}$, the parameter space is compact, and $\ell_{0}(\gamma)$ is continuous, then the estimator $\widehat{\gamma}$ obtained by maximizing $\widehat{\ell}(\gamma)$, converges in probability to $\gamma_{0}$, that is, $\widehat{\gamma}\overset{p}{\longrightarrow}\gamma_{0}$.

For $\mathscr{L}_{\scalebox{0.5}{\text{W}}}(\gamma)$, the sample size-normalized objective function to be maximized is

[TABLE]

Because $c$ is assumed to be bounded away from 0, and provided the technical conditions for the uniform weak law of large numbers obtain, the above objective function converges uniformly in probability to

[TABLE]

We will now show that $\ell_{0}(\gamma)=\ell_{\scalebox{0.5}{\text{W}}0}(\gamma)$.

By design, if $S=1$, then $SD=1$; if $S=0$, then $SD=0$. Thus, to establish the result we only need to show that

[TABLE]

Starting from the right-hand-side,

[TABLE]

which establishes the result.

Because $\ell_{0}(\gamma)=\ell_{\scalebox{0.5}{\text{W}}0}(\gamma)$, it follows that the maximizer of $\widehat{\ell}_{\scalebox{0.5}{\text{W}}}(\gamma)$, $\widehat{\gamma}_{\scalebox{0.5}{\text{W}}}$, converges in probability to $\gamma_{0}$, that is, $\widehat{\gamma}_{\scalebox{0.5}{\text{W}}}\overset{p}{\longrightarrow}\gamma_{0}$.

To obtain the asymptotic distribution of the estimators, we need additional technical conditions as in Theorem 3.1 of [17]; provided these conditions hold, $\widehat{\gamma}$ and $\widehat{\gamma}_{\scalebox{0.5}{\text{W}}}$ are asymptotically normal.

Appendix C Nested trial design with covariate-dependent sampling probabilities

C.1 Sampling properties

As noted in the main text, a more general version of the nested trial design assumes that the sampling probabilities for non-randomized individuals depend on baseline auxiliary covariates. Let $X=(X_{1},X_{2})$, where $X_{1}$ represents baseline auxiliary covariates that are available on all members of the actual population in which the trial is nested, and $X_{2}$ represents covariates that are only measured among randomized individuals ($S=1$) and sampled non-randomized individuals ($S=1,D=1$).

The identifiability conditions and identification results remain the same as in the main text; but the sampling properties of this design are

[TABLE]

where $0<c(X_{1})\leq 1$ is a known function that only depends on $X_{1}$, allowing the sampling of non-randomized individuals to depend on the auxiliary covariates that are available from all members of the actual population.

C.2 Identification of the conditional probability of trial participation

Using an argument similar to the case when the sampling probability for non-randomized individuals does not depend on covariates, we obtain

[TABLE]

which is identifiable because the inverse of the conditional odds of trial participation in the sampled data, $\dfrac{\Pr[S=0|X,D=1]}{\Pr[S=1|X,D=1]}$, are identifiable, and $c(X_{1})$ is known, by design.

C.3 Estimating the probability of trial participation by weighted regression

As before, we assume a model $p(X;\gamma)$ for $\Pr[S=1|X]$ with finite-dimensional parameter $\gamma$. The weighted pseudo-likelihood function becomes

[TABLE]

Note that the only difference between $\mathscr{L}^{*}_{\scalebox{0.5}{\text{W}}}(\gamma)$ and $\mathscr{L}_{\scalebox{0.5}{\text{W}}}(\gamma)$ is that the weights in the former depend on $X_{1}$. The sample size-normalized objective function to be maximized is

[TABLE]

Because $c(X_{1})$ is assumed to be bounded away from 0, and provided the technical conditions for the uniform weak law of large numbers obtain, the above objective function converges uniformly in probability to

[TABLE]

We will now show that $\ell_{0}(\gamma)=\ell^{*}_{\scalebox{0.5}{\text{W}}0}(\gamma)$.

As noted above, by design, if $S=1$, then $SD=1$; if $S=0$, then $SD=0$. Thus, to establish the result we only need to show that

[TABLE]

Starting from the right-hand-side,

[TABLE]

which establishes the result.

Because $\ell_{0}(\gamma)=\ell^{*}_{\scalebox{0.5}{\text{W}}0}(\gamma)$, it follows that the maximizer of $\widehat{\ell}^{*}_{\scalebox{0.5}{\text{W}}}(\gamma)$, $\widehat{\gamma}^{*}_{\scalebox{0.5}{\text{W}}}$, converges in probability to $\gamma_{0}$, that is, $\widehat{\gamma}^{*}_{\scalebox{0.5}{\text{W}}}\overset{p}{\longrightarrow}\gamma_{0}$.

In practical terms, this result suggests that the conditional probability of trial participation in the target population can be estimated using a weighted regression of $S$ on $X$ among sampled patients, using weights equal to 1 for randomized patients (all of whom are sampled); $1/c(X_{1})$ for sampled non-randomized individuals; [math] for non-sampled non-randomized individuals.

As above, provided the technical conditions of Theorem 3.1 of [17] hold, $\widehat{\gamma}^{*}_{\scalebox{0.5}{\text{W}}}$ is asymptotically normal.

\ddmmyyyydate\newtimeformat

24h60m60s\twodigit\THEHOUR.\twodigit\THEMINUTE.32 \settimeformat24h60m60s

transportability_study_design,
Date:    \currenttime Revision: 31.0