Test-Negative Designs with Multiple Testing Sources

TL;DR

This paper discusses how test-negative designs can be used to evaluate vaccine efficacy when individuals are tested for different reasons, including both symptomatic and asymptomatic cases.

Contribution

The paper introduces a method to estimate vaccine efficacy using multiple testing sources in the context of an Ebola vaccine trial.

Findings

A simple approach is proposed to estimate vaccine efficacy using two sources of test positives and test negatives.

The method also assesses whether vaccine efficacy is consistent across different testing sources.

The approach is relevant for future trials, even though the specific Ebola trial was not completed.

Abstract

Test-negative designs, a form of case-cohort studies, have been commonly used to assess infectious disease interventions. Early examples of the design included the evaluation of seasonal influenza vaccines in the field. Recently, they have also been widely used to evaluate the efficacy of COVID-19 vaccines in preventing symptomatic disease for different variants1. The design hinges on individuals being tested for the disease of interest; upon recruitment, such individuals are subjected to a definitive test for the presence of the disease of interest (test-positives), or not (test-negatives) along with the determination of whether the individual has been exposed to the intervention under study (e.g. vaccination). In most early TND studies, individuals were tested because they were suffering from symptoms consistent with the disease in question, and the TND was a tool to reduce confounding due to healthcare-seeking behavior. However, in many cases, such as COVID-19, and Ebola, testing results were available at healthcare facilities for individuals who presented for a variety of reasons in addition to symptoms (e.g., case contact tracing, etc.). Aggregating samples from symptomatic and asymptomatic test results leads to bias in the assessment of the efficacy of the intervention. Here we consider these issues in the context of a specific version of the ‘multiple reasons for testing problem,’ motivated by a vaccine trial designed to assess a new Ebola viral disease vaccine (EVD)2. Some participants are recruited in the usual TND fashion as they present for care suffering from symptoms consistent with an Ebola diagnosis (and are thus tested); in addition, however, any test-positive identified in this fashion leads to immediate testing for Ebola for all close contacts of the test-positive who are likely asymptomatic at that point. We examine a simple approach to estimate the common efficacy of the vaccine intervention based on these two sources of test positives and test negatives, complemented by an assessment of whether efficacy is the same for both sources. The EVD trial was not completed for the fortunate reason that the prevailing disease outbreak ended; nevertheless, the approach here will be important if this trial is ever recommenced or similar trials are conducted in the future

Figures

Full text

INTRODUCTION

1 |

The test-negative design (TND) has become common in studies of infectious disease interventions, and many TND studies have informed policy in the past decade. For example, policies informed by TND studies have included annual seasonal influenza vaccine recommendations^3^, implementation of a three-dose pneumococcal conjugate vaccine (PCV) schedule in the UK and other countries^4^, recommendations to remove upper age restrictions for receipt of rotavirus vaccines among children, and administration of oral cholera vaccines as a single dose during campaigns in emergency settings^5,6,7^.

The TND can be viewed as a version of a case-cohort study in which symptomatic patients meeting predefined inclusion/exclusion criteria are enrolled and subsequently classified as test-positive “cases” or test-negative “controls” based on the results of definitive diagnostic testing for the outcome under study. At the same time, eligible participants are classified as exposed or unexposed to an intervention of interest such as vaccination. As noted, this design is frequently used for evaluating the effectiveness of seasonal influenza vaccination^8,9,10,3^, and its internal validity has been explored in depth^11,8,12,13,14,10,15^. Briefly, validity depends primarily upon the avoidance of selection bias in the sampling of cases and controls, as well as the the extent to which the exposure distribution among controls is representative of the exposure distribution among the source population that gives rise to cases. Key assumptions and their relevance to test-negative studies are discussed briefly below. For ease of discussion we refer to the outcome of those testing positive for the pathogen of interest as “test-positive illness” and in those testing negative as “test-negative illness.”

The key assumptions for unbiased estimation of an intervention effect using data from a standard TND study include that (i) test-negative illness occurrence is not associated with the intervention, (ii) the relative propensity of individuals to seek health care is non-differential by outcome status, (iii) the test used to determine disease status is highly sensitive and specific, i.e. no misclassification, (iv) the sampling of test-negatives is unfiltered; that is, controls must be sampled from the whole population at risk without excluding those who test-positive at another time during the study period, (v) the efficacy of the intervention is not modified by healthcare-seeking behavior, and (vi) participants with test-negative illness are recruited only when test-positive illness is circulating. The first of these assumptions requires a careful definition of what is considered to be a suitable test-negative illness that also must exhibit non-specific symptoms similar to those that occur with the test-positive illness. Assumption (ii) is plausible if tested individuals are not aware of their (likely) outcome status prior to seeking care and testing. Assumption (iii) can be weakened by appropriate statistical adjustments. Assumptions (iv) and (vi) guarantee that test-negative controls are from the source population at risk for the test-positive illness. Finally, assumption (v) allows extrapolation from the healthcare-seeking population to the general population.

Because cases (test-positives) and controls (test-negatives) are recruited from the same patient population, and restricted to those seeking care, the design was intended to ameliorate if not eliminate, bias caused by health-care-seeking behavior^8,14,10^.

We continue our discussion with a definition of the causal estimand of interest for vaccine effectiveness studies, namely the (causal) population Relative Risk ( $[eqn]$ ) defined as the ratio of the incidence proportions of the disease of interest comparing the population under the two counterfactual conditions, the first where all members of the population are vaccinated and the other where no one is vaccinated. It is important to note here the specific definition of incidence being used. For the target estimand for a TND, we define incidence as both testing positive for the pathogen of interest and developing symptoms requiring health care. For convenience, we shall denote this true comparative population $[eqn]$ by $[eqn]$ , again with the understanding that this efficacy is measured by a reduction in the likelihood of developing the specified symptoms necessary for inclusion. In Section 2, we also consider including asymptomatic infections (i.e. test-positives).

Several authors have explored the statistical rationale and underlying assumptions of the traditional TND, showing that the empirical odds ratio $[eqn]$ —i.e., the observed sample odds of exposure in test-positives relative to that for test-negative controls—targets $[eqn]$ the population Relative Risk ( $[eqn]$ ), in the healthcare-seeking population, providing underlying assumptions (noted above) are met^8,14^. This is a direct consequence of the case-cohort nature of the sampling of test-positive and test-negative participants. We emphasize again that this definition of Relative Risk is specific to the definition of a test-positive outcome in the sense that it depends on the level of symptoms that are required to be eligible for recruitment. As such, this Relative Risk may vary if eligibility requires more serious symptoms such as hospitalization, etc.

This causal estimand of interest as defined by potential outcomes ( $[eqn]$ ) can be equivalently assessed through the use of directed acyclic graphs as long as the standard causal assumptions of (i) consistency, (ii) no interference, and (iii) no unmeasured confounding are satisfied. The last of the these three assumptions requires that all relevant confounding variables are included in the relevant causal diagram. The causal structure underlying this approach has been studied in detail^10^, with a particular focus on the fact that health care seeking behavior may not be a binary characteristic. For the traditional TND examining influenza, exposure refers to seasonal vaccination so that an estimate of the RR for the outcome associated with exposure directly yields an estimate of vaccine effectiveness, 1 – $[eqn]$ .

Throughout the statistical development described above, it has been assumed that both test positives and test negatives are ascertained through healthcare facilities where potential participants seek care because of undifferentiated symptoms so that such individuals are blinded to their disease status at the time of seeking care when their exposure status (i.e. vaccination) is also determined. However, in recent applications, this simple ascertainment process is not satisfied in that there may be multiple reasons why individuals seek testing (that ultimately determines whether they are classified as test-positives or test-negatives). For example, in recent Covid-19 test-negative studies^16^, individuals may seek, or be referred for, asymptomatic testing for screening purposes (for travel, etc), or because these individuals may have been exposed to the virus (by a family member, for example).

^17^ pointed out that reasons for testing other than symptoms may introduce bias, and^18^ conceptualized this issue through a causal graph. Both papers, and^19^, state that a stratified analysis based on testing reasons could alleviate the bias, but no quantification or formal solution was given.

Here we focus on a specific example of an extended test-negative design with two distinct sources of participants, motivated by a proposed study to assess a vaccine for Ebola virus disease (EVD) in the eastern Democratic Republic of Congo^2,20,21,22^. In this trial, a traditional test-negative design was employed but, in addition to traditional symptomatic testing, close contacts of any observed test-positives in the first stage were tested for the presence of the Ebola virus, prior to the onset of any symptoms. Vaccination status was ascertained for both symptomatic and asymptomatic participants at the time of recruitment.

^23^ quantify the bias from a naive TND analysis based on data from this kind of hybrid design with two distinct reasons for testing: (i) symptomatic self-reporting or (ii) testing close contacts of known cases, and proposed a weighted average estimator to remove bias from naive aggregation. This required a strong assumption that the VE for either recruitment population is the same, an assumption we examine further below.

A naive solution to filter out asymptomatic subjects and only use the remaining data to perform classical TND analyss was also discussed. Although this procedure is valid, it loses a large amount of information, yielding less efficient estimation. In this paper, we discuss the analysis of data from this hybrid design and examine a simple method for assessing the shared efficacy of vaccine intervention in response to these two sources (symptomatic samples and case contact samples) of test positives and test negatives, together with an assessment of whether the efficacy is the same for both sources. The merits of our method are shown by comparing results with existing benchmark estimators.

TRADITIONAL TEST-NEGATIVE DATA SUPPLEMENTED BY TESTING OF CLOSE CONTACTS OF CASES

2 |

Before discussing the causal structures underlying the recruitment of both symptomatic and asymptomatic participants, we discuss briefly the role of an intervention such as vaccination in both preventing infections and specific symptoms. Following^17^, let $[eqn]$ be the relative risk of infection for a vaccinated versus unvaccinated individual, resulting from a reduced susceptibility to the acquisition of a pathogen or accelerated clearance of the pathogen. Again, this causal parameter can be defined in terms of potential outcomes. Infection here is measured through a highly accurate diagnostic test even if no symptoms ever present. Further, define $[eqn]$ as the relative risk of the infection meeting a particular severity threshold, e.g., clinical symptoms requiring care, hospitalization, intensive care unit admission, or mechanical ventilation) for an infected vaccinated individual versus an infected unvaccinated individual (where $[eqn]$ stands for progression), again using potential outcomes to stress the causl nature of the estimand.

Then the overall relative risk (for the pre-defined symptom severity outcome) associated with vaccination is just $[eqn]$ , with $[eqn]$ . As we discuss below, the traditional TND (with symptomatic recruits) estimates $[eqn]$ but cannot separate out the distinct vaccine effects $[eqn]$ and $[eqn]$ . Further, this Relative Risk is estimated only in the healthcare-seeking population from which participants are drawn absent assumption (v) discussed in the previous section–for clarity, we refer to this Relative Risk as $[eqn]$ . However, as we elucidate below, testing of case contacts provides direct estimates of $[eqn]$ —which may differ from $[eqn]$ —in the entire population, whether healthcare-seekers or not. In particular, $[eqn]$ only if $[eqn]$ , that is, the intervention—e.g. vaccination—has no influence on symptoms after infection has occurred. In short, for $[eqn]$ , we require—at least—assumption (v) above and that $[eqn]$ . We discuss additional potential confounding below.

We now elucidate this reasoning by exploiting two Directed Acyclic Graphs (DAGs) in Figure 1 associated with test-negative designs for the two distinct reasons for testing.

Test-Negative Design with Symptomatic Participants

2.1 |

The illustration on the left panel of Figure 1 describes a DAG for a traditional test-negative design that recruits symptomatic patients only. These are individuals who proactively seek medical attention when they experience symptoms pertinent to the disease of concern (and the test-negative conditions for that matter). Within this population, the results of an appropriate diagnostic test are classified into two groups: ‘test-positive’ and ‘test-negative’ (control).

We define the several nodes, or variables, within this DAG, as follows:

$[eqn]$ : This node denotes the variable associated with quantifying healthcare-seeking behavior. For simplicity, we treat this variable as binary. Specifically, $[eqn]$ indicates that the participant seeks care upon onset of the symptoms defined by the study protocol. Only such individuals can be recruited in a traditional test-negative study.V: This node refers to the vaccination, or intervention, status of the patient.D: This node signifies the disease status, elucidating whether the target infection has occurred or not (i.e., a test-positive result).Symp: This node is an indicator variable for the presence of disease-related (test-positive or test-negative) symptoms. $[eqn]$ : This depicts whether a test has been conducted, essentially indicating that the participant is eligible and has been recruited for the study.

The arrow $[eqn]$ is indicative of the fact that a patient’s vaccination status can influence the likelihood of contracting the disease in question. There is no direct effect from $[eqn]$ since taking the test is entirely determined by health-seeking behavior and symptoms.

Next, we introduce an additional assumption regarding the relationship between vaccine effectiveness, healthcare-seeking behavior, and other (test-negative) illnesses that permit identification.

Assumption 1.

The effectiveness of the vaccine is not modified by healthcare-seeking behavior, and the vaccine status does not cause the presence (or not) of test-negative illnesses.

The second part of this assumption (discussed as Assumption (i) in Section 1) permits the identification of vaccine effectiveness, whereas the first part allows extrapolation of estimated vaccine effectiveness to the entire symptomatic population whether they seek care or not. These assumptions are also made in many previous test-negative design papers^8,23^. Note that, as discussed above, the traditional test-negative design cannot assess vaccine effectiveness in preventing infection, only on preventing symptomatic infection. We return to this later when we expand the design to include testing asymptomatic participants. Identification of vaccine effectiveness within the symptomatic tested population then follows:

$$[eqn]$$

There are several steps listed above that merit further explanation. Specifically, equality $[eqn]$ holds since

$$[eqn]$$

Essentially, with a given vaccine status $[eqn]$ , $[eqn]$ , and healthcare-seeking behavior $[eqn]$ , the decision to test $[eqn]$ remains independent of the disease status $[eqn]$ . This is a valid observation in the context of the symptomatic healthcare-seeking population, where the decision to pursue testing is primarily influenced by healthcare-seeking behavior, symptoms, and vaccine status. Disease status only influences testing behavior indirectly, through the manifestation of relevant symptoms. Note that in well-designed test-negative studies, this assumption is reasonable since participants do not know their exact disease status before testing. The assumption might be violated if, for example, other circumstances suggest disease status to the participant before testing (such as knowledge of close contact with the relevant pathogen of interest).

Equality $[eqn]$ holds since

$$[eqn]$$

The underlying logic of this equation is that given a certain health-seeking behavior $[eqn]$ and disease outcome $[eqn]$ , the status of symptoms is directly dictated by the presence of disease $[eqn]$ , and remains unaffected by vaccine status. This directly assumes that the vaccine does not prevent (or ameliorate) symptoms, or in symbols that $[eqn]$ ; if infected by the test-positive pathogen, it is possible that prior vaccination may reduce or even eliminate symptoms (as has been postulated for Covid-19 vaccines). This is unlikely to be true for vaccines for Ebola viral disease, however.

Equality $[eqn]$ is valid since per Assumption 1, amongst control participants (i.e. test-negatives), the vaccine status is independent of symptoms, and its relative frequency reflects that of the general healthcare-seeking population.

Finally, Assumption 1 posits that the result of the intervention is not modified by the health-seeking behavior, validating Equality $[eqn]$ . Consequently, we derive the following expression:

$$[eqn]$$

Based on the identification assumptions, we are thus able to leverage the symptomatic participant data on vaccination to ascertain the true effectiveness of the vaccine. We now briefly outline the sampling context for the traditional test-negative design used for recruiting individuals. Table 1 provides the breakdown of both test-positives $[eqn]$ and test-negatives $[eqn]$ by their vaccination status. In this table, $[eqn]$ denotes the observed number of vaccinated individuals in the test-positive cases in the test-negative design sample, etc. This yields the following (conditional) likelihood, where we use the subscript $[eqn]$ to denote the data arising from symptomatic recruited patients:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ with $[eqn]$ . Note that, from above, we do not need to condition on $[eqn]$ by Assumption 1; also, $[eqn]$ does not condition on $[eqn]$ due to the case-cohort nature of the sampling; that is, controls may also be sampled as cases. In this context, $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ represent the number of observed test samples in distinct categories from symptomatic participants.

Testing of Contacts of Positives (Cases)

2.2 |

The right-hand panel of Figure 1 illustrates a Directed Acyclic Graph (DAG) for data that arises from testing close contacts of test-positives. In this population, the testing behavior, as per the DAG, is solely determined by the exposure status $[eqn]$ , where $[eqn]$ indicates that an individual is a close contact with a known person who has tested positive for the presence of $[eqn]$ ; the decision to test is unaffected by the individual’s vaccine status $[eqn]$ and disease status $[eqn]$ , given $[eqn]$ . Such a DAG might describe scenarios involving severe diseases, such as Ebola, where individuals who have had close contact with confirmed positive cases are expected to undergo immediate (asymptomatic) testing. Consequently, we make the following assumption:

Assumption 2.

Any individual identified as a close contact is obligated to undertake a screening test for the presence of $[eqn]$ , that is,

$$[eqn]$$

This assumption follows the protocol of the Ebola vaccine trial^23^.

Next, we propose an additional assumption that the efficacy of the vaccine is independent of the exposed population.

Assumption 3.

The effectiveness of vaccine is identical among the exposed and unexposed (on the risk ratio scale).

This assumption is not about potential confounding but homogeneity of effect, stating that the causal relative risk amongst contacts of known cases does not differ from the analogous causal relative risk in individuals outside of the study. Note that such individuals who become infected must have experienced a case contact at some point. In addition, in most vaccine studies, there is often little evidence of vaccine effectiveness varying by population subgroups (for Covid-19, the original trials of the Pfizer and Moderna vaccines considered subgroup analyses). A major exception to this general statement concerns immunocompromised patients where vaccines often provide weaker protection, a situation where there is a strong biological rationale to expect a difference. This comment also supports the first part of assumption 1 in Section 2.1 regarding homogeneity of vaccine effectiveness across differing healthcare-seeking subgroups.

Under these two assumptions, and using $[eqn]$ to denote the relative frequency of positive test results among the tested contacts, we have

$$[eqn]$$

Equality $[eqn]$ follows from Assumption 2, which posits that all close contacts undergo testing given exposure to confirmed positive cases. Furthermore, Equality $[eqn]$ is valid due to Assumption 3, which assumes that the efficacy of the vaccine is independent of exposure status. Note that, in this context, the variable $[eqn]$ requires only a positive test for infection with the pathogen and does not require the presence of any symptoms. Recall that $[eqn]$ if $[eqn]$ .

Given this identifiability, we consider the following (conditional) likelihood arising from Table 2 data that classifies contacts’ test results by outcome and vaccination status of the contact. In this table, $[eqn]$ counts the number of exposed contacts who both test positive at recruitment and are vaccinated, etc.

$$[eqn]$$

where $[eqn]$ and $[eqn]$ .

There is a tacit assumption made here that any subsequent intervention on contacts of known cases does not affect their chances of infection as measured at the time of recruitment (although it might improve symptoms after any infection). Further, we assume temporality in that contacts who test-positive are assumed to have followed infection of the case and not the reverse, this assumption being based on the earlier appearance of symptoms in the case.

Combined Likelihood

2.3 |

From the discussion of Sections 2.1 and 2.2, we note the presence of two distinct likelihood functions, each capable of evaluating the effectiveness of the vaccine using observed samples, albeit estimating somewhat different measures of efficacy absent further assumptions. If we invoke the assumptions discussed above, including that $[eqn]$ , and we assume that the two estimands are equivalent for this and other reasons (we turn to a discussion of confounding later in this section), we propose a straightforward methodology to assess the collective efficacy of the vaccine intervention by aggregating the two distinct sources of information. If we use a composite likelihood, the full (conditional) likelihood is then simply $[eqn]$ . When contacts of cases from the initial TND are tested (as was planned for the motivating Ebola trial), the use of this combined likelihood makes an additional assumption, namely that the outcome of a case contact is independent of the characteristics of the associated case. This assumption can be weakened if covariates are introduced into the analysis (as discussed further below).

The negative logarithm of this combined likelihood, $[eqn]$ simplifies to

$$[eqn]$$

based on the data from Tables 1 and 2. Furthermore, from sections 2.1 and 2.2, we substitute $[eqn]$ with $[eqn]$ and replace $[eqn]$ with $[eqn]$ . Our primary interest lies in conducting statistical inferences for the common $[eqn]$ .

We note that the parameters $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ are constrained within the interval [0, 1], whereas the parameter $[eqn]$ is confined to the non-negative real numbers. These restrictions pose challenges for optimization procedures due to the necessity of operating within a constrained parameter space. To address this, we use a simple reparameterization. Specifically, we redefine $[eqn]$ as $[eqn]$ , $[eqn]$ as $[eqn]$ , and $[eqn]$ as $[eqn]$ .

Then, employing this reparameterization in the combined likelihood (3) yields

$$[eqn]$$

Subsequently, assuming that the true value of ( $[eqn]$ , $[eqn]$ , $[eqn]$ ) does not lie on the boundary of the parameter space, it is then straightforward to establish the existence of a unique maximum likelihood estimate, denoted by $[eqn]$ .

We end this section by noting that, of course, the two likelihood functions (1) and (2) can be maximized separately to provide estimates of $[eqn]$ and $[eqn]$ . With this approach, a likelihood ratio test can be used to test the equality of the two underlying Relative Risks. Note that rejection of the null hypothesis, in this case, does not formally establish that $[eqn]$ since there are other potential explanations of why the two Relative Risks may differ. For example, the exposure experiences of close contacts of cases may differ from those recruited symptomatically from the general population which may affect the performance of a vaccine intervention. Further, there is the issue of confounding by healthcare-seeking behavior amongst case contacts. Nevertheless, this likelihood ratio test may provide some evidence that a vaccine influences the progression of symptoms after breakthrough infections.

To address potential confounding, it is important to expand the likelihood functions (1) and (2) to incorporate the influence of relevant covariates. The choice of covariates might differ for the distinct likelihoods as varying covariate information might be available from the two data sources. Particularly important here is the role of healthcare-seeking behavior. This variable is naturally controlled in the estimation of $[eqn]$ from the first likelihood (since sampling is restricted to healthcare-seekers by design), $[eqn]$ , but is not addressed by naive use of $[eqn]$ . In this case, measurement and adjustment for covariates that act as proxies for healthcare-seeking behavior may be especially important. When covariates are included (through the use of appropriate generalized linear models (GLMs) in both likelihood components), the estimand reflects a conditional Relative Risk and thus vaccine effectiveness. Given the collapsibility of the Relative Risk, this will still yield a marginal Relative Risk associated with $[eqn]$ so long as there is no interaction between $[eqn]$ and any of the included covariates. Note that—for most vaccines—there is usually no biological basis for expecting such interactions to occur (other than perhaps for measures of immune compromise).

Of further interest is the vaccination status of the case associated with contacts in data underlying likelihood function (2). Information on the vaccination status of the assumed infected person (the individual who causes the infection, or not, in a contact) should be available for contacts of known cases (particularly if the cases arise from symptomatic recruitments in the underlying TND study). It is unlikely, however, that this is known for the original TND participants since, in most cases, the source of their infection is likely unknown. Analysis of this covariate in an expanded version of (2) would allow an assessment of whether vaccination reduces infectiousness in a vaccinated individual who experiences a breakthrough infection.

Note that adjustment for covariates may also control for any source of dependence between a case (used in likelihood (1)) and contacts (used in likelihood (2)) which has been ignored in use of the combined likelihood as noted above.

NUMERICAL STUDIES

3 |

In this section, we assess the performance of the combined maximum likelihood estimator through simulations. For simplicity, we consider sample sizes of $[eqn]$ with $[eqn]$ ^‡^.

We selected the true parameters $[eqn]$ and $[eqn]$ while varying the value of $[eqn]$ across the range 0.6 ∼ 1.0, reflecting high rates of vaccination and moderate values of efficacy. We provide a detailed implementation of the optimization procedure in the appendix.

We compare the performance of the estimator $[eqn]$ , obtained by minimizing the combined loss function in (3), with additional benchmark estimators. In simulating the data, we allow for two situations: (i) the vaccine effects of the two sources are the same, and (ii) the vaccine effects from the two sources are different, for example, whether $[eqn]$ or not.

For each case, we study four benchmarks involving different weighted averages of the two estimators based on separate maximizations of (1) and (2), denoted as $[eqn]$ , where $[eqn]$ and $[eqn]$ . Here, $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ represent the separate maximum likelihood estimators based on (1) and (2), respectively. It is noteworthy that a weighted estimator was previously suggested by^23^; however, the specific choice of weights was not explicitly discussed. Therefore, we compare estimation results using $[eqn]$ with (i) the traditional TND estimator where we set $[eqn]$ (benchmark 1) and (ii) a simple average (with $[eqn]$ , $[eqn]$ ) (benchmark 2) and (iii) a third benchmark (benchmark 3) where the weights are proportional to the (estimated) inverse standard errors of $[eqn]$ and $[eqn]$ . In addition, we also compare $[eqn]$ with an odds ratios estimator based on a naive aggregation of the two distinct testing samples from the two sources (the fourth benchmark); that is, the estimator $[eqn]$ .

For each assumed value of $[eqn]$ , we generate 500 simulated data sets, reporting the average value of our estimator $[eqn]$ as well as the standard error of $[eqn]$ . Detailed simulation results are presented in Table 3. We also consider scenarios where the vaccine effects from the two testing sources are different (e.g. when $[eqn]$ ). In this case, we study the performance of the aforementioned estimators and also conduct a likelihood ratio test with significance level $[eqn]$ on the equivalence of the vaccine effect from two sources. The results are presented in Tables 4 and 5, respectively.

Results

3.1 |

Based on Table 3, the simple aggregated odds ratio (without differentiating the testing sources/reasons; that is, Benchmark 4) exhibits bias. This occurs even though the vaccine effects from the two distinct testing sources are identical, leading to a failure to accurately identify the true vaccine effect. In contrast, when data from the two testing sources are separately analyzed, our proposed estimator—derived from the combined likelihood of Equation 3—demonstrates greater efficiency as compared to the other benchmark estimators. This assertion is further corroborated by the theoretical findings presented in the appendix, wherein we establish that the variance of the proposed estimator achieves the Cramér–Rao lower bound.

Table 4 reveals that when the vaccine effects from the two testing sources differ $[eqn]$ , all combined estimators exhibit bias. Note that the simulations here reflect a possible scenario where $[eqn]$ , so the vaccine here does not reduce the risk of infection but reduces the risk of symptoms post-infection. Therefore, in reality, it is important to first assess the comparability of vaccine effects from the distinct data sources, for example by implementing a likelihood ratio test. To this end, we present the results of conducting such a likelihood test in Table 5. When the null hypothesis holds, the type I error is well controlled under 5%, and when the alternative holds, the power rapidly increases to 1 when the number of testing samples grows.

In principal, contacts of known cases for whom infection is observed could be followed to detect onset of symptoms. This would allow a direct assessment of whether the vaccination status of an infected individual affects symptom development. However, in practice, such an assessment is not likely to reflect the role of vaccination in general as other interventions that may influence symptoms will likely be available to infected contacts immediately upon determination of their infection.

DISCUSSION

4 |

Data from the motivating Ebola vaccine trial is not available for analysis for the very best of reasons, namely that the 2018 Ebola outbreak in the Democratic Republic of the Congo ended. Nevertheless, the methodology developed here will be valuable if the vaccine trial is ever reopened, or if similar vaccine trials with this design are planned in the future. We have shown here that there is a gain in efficiency in using the hybrid design although, as with any observational study, and the traditional test-negative design in particular, this comes with the requirement of additional assumptions. The hybrid design also permits careful consideration of whether a vaccine only ameliorates symptoms or whther it also protects against infection.

Our methodological assessment of the motivating Ebola vaccine study example with two sources of testing data illustrates the importance of considering the reasons for testing and avoiding any temptation to aggregate testing data naively. Further, it is crucial to understand that data arising from different testing sources may estimate different measures of effectiveness. The design of such hybrid TND studies must carefully consider an appropriate discussion of symptoms for eligibility as well as the possibility that the intervention may alleviate the development of such symptoms separately from any effect on infection. In most cases, it will also be important to pre-define covariates for measurement that will allow adjustment for potential confounding by healthcare-seeking behavior when implementing asymptomatic testing. When the estimands associated with different testing sources differ, further information will usually be required to disentangle the root causes of such differences.

Supplementary Material

Supplement 1