Modelling approaches for estimating vaccine effectiveness of consecutive SARS-CoV-2 variant sublineages in the absence of study-specific genetic sequencing data, VEBIS hospital network, Europe, 2023/24

TL;DR

This paper introduces new methods to estimate vaccine effectiveness against SARS-CoV-2 variants when genetic sequencing data is unavailable.

Contribution

Two novel statistical approaches (VCM and VPM) are proposed to estimate vaccine effectiveness without requiring study-specific genetic sequencing.

Findings

VE estimates using VCM and VPM were within ±3% of standard methods for XBB variant.

VPM performed well even with limited data, unlike sequencing-based validation.

VE against BA.2.86 was lower than against XBB using both models.

Abstract

Genetic changes in COVID-19 variants/sublineages (VSLs) can reduce vaccine effectiveness (VE). Timely VSL-specific VE estimates are essential, but study-specific VSL identification by whole genome sequencing (the “gold standard”) is expensive and time-consuming. Alternatively, VSL-specific VE has been estimated from external sequencing data (VSL predominance period by proxy: PP). We propose two novel approaches for use in test-negative design (TND) studies to estimate VSL-specific VE when study-specific VSL identification is not possible.

We demonstrate the variant category model (VCM) and the variant proportion model (VPM) approaches. Using data from a hospital-based TND study among adults ≥65 years, during the period of sequential predominance of XBB and BA.2.86 in 2023/24, we estimated the VE as (1-OR) x 100%. For the VCM, we used a binary variable categorising “most likely underlying sublineage” based on publicly available sequencing data. For the VPM, we used a continuous variable with values from 0 to 1 representing the weekly proportion of BA.2.86. We validated results using study-specific VSL identification from sequenced study data (SD) and the standard PP approach.

Overall, at 14–59 days post vaccination, VE point estimates against XBB were within ±3% absolute for the VE estimated using both models, with an equivalent standard PP validation. We could not validate using SD, as there were no vaccinated XBB cases. Against BA.2.86, VE was lower than against XBB, and the VCM and VPM results were within ±7% absolute of each other, with lowest validation results from SD but equivalent results from the PP.

Both proposed approaches produced similar VE estimates to those from well-known methods. The VPM could also provide VE estimates when the validation techniques were limited by low sample size.

Figures

Demonstration of the study period definitions for the predominance period approaches.(a) standard PP VE against XBB VSL, (b) standard PP VE against BA.2.86 and novel model approaches: (c) the VSL category (0 if XBB and 1 if BA.2.86) model and (d) VSL proportion (prevalence of BA.2.86 VSL) model. VEBIS hospital network, Europe. Standard PP: predominance period approach – standard technique using proxy for sequenced data; VCM: novel VSL category model; VEBIS: Vaccine Effectiveness, Burden and Impact Studies; VPM: novel VSL proportion model; VSL: variant/sublineage.

Effectiveness of the monovalent XBB.1.5 COVID-19 vaccines against hospitalisation among individuals aged ≥65 years old, by BA.2.86 VSL proportion and by time since vaccination, using the four approaches: the gold standard sequencing data (gold standard SD), standard predominance period (standard PP), adapted predominance period with VSL category model (PP VCM) and with the proportion model (PP VPM); VEBIS hospital network, Europe.

Comparison of the effectivenessa of the monovalent XBB.1.5 COVID-19 vaccines against XBB and BA.2.86 VSLs among hospitalised SARI patients aged ≥65 years, by BA.2.86 VSLs using adapted PP with VPMb, by TSV, VEBIS hospital network, Europe.OR: odds ratio; PP: standard predominance period; TSV: time since vaccination; VE: vaccine effectiveness; VEBIS: Vaccine Effectiveness, Burden and Impact Studies; VPM: VSL proportion model; VSL: variant/sublineage. a The comparison of the VE is measured by the ratio of the OR of vaccination between cases and controls against BA.2.86 and XBB VSLs (ORBA.2.86/ORXBB), for the proportion of circulating BA.2.86 VSLs ≥ 60%. The ratio of OR is then calculated as ORBA.2.86/ORXBB=ORp/OR1−p, where p is the proportion of BA.2.86 VSLs. b The VPM model is a logistic regression model having case/control status as the dependent variable and the date of symptom onset, age, sex, presence of a chronic condition, vaccination status, VSL proportion and an interaction term between vaccination and the VSL proportion as independent variables. The form used to include date of symptom onset (linear, week, month, splines) and age (10-year age group, linear, splines) in the model was selected as the model with the lowest AIC.

Equations

Full text

Introduction

Genetic changes in SARS-CoV-2 have been shown to affect viral characteristics such as transmissibility, virulence, and immune evasion, as well as therapeutic response of emerging SARS-CoV-2 variants [1,2]. The emergence of new variants/sublineages (VSLs) may result in reduced vaccine effectiveness (VE), increased risk of disease severity (particularly among older and other at-risk populations [1]), and compromised healthcare systems. VSL-specific VE estimation is key for the development of vaccines to ensure continued protection as the virus evolves, particularly among those most vulnerable. Timely VSL-specific VE estimates may inform immunisation programmes to be adapted and mitigate the impact of the newly emerging variants.

There are three ways to monitor existing, and identify emerging variants, of which the gold standard is whole genome sequencing [3]. Methods to estimate VE against a VSL of interest previously relied on study-specific VSL identification using either sequencing data from a specimen collected from study participants, linked with epidemiological data [4,5] or inferred from RT-PCR techniques [6,7]. Although most VE studies require PCR confirmation of cases, VSL identification is not a requirement for patient treatment. Adding a sequencing component to VE studies requires additional efforts in terms of logistics and cost, and can impact the timeliness of results [4]. In addition, linkage of general surveillance sequencing to study data, particularly for studies collecting primary data, can be difficult. Sequencing data may not be available when VE estimates are prepared and, even when available, if there is only a low number of positive samples sequenced, this can limit precision of the VSL-specific VE estimates [4]. During the COVID-19 pandemic period, some studies began using a proxy to genetically classify viruses and VSLs, by restricting the study period to when the virus or VSL of interest was predominant (often with varying predominance thresholds), based on external data sources [8–13].

During the 2023/24 COVID-19 vaccination campaign in the European Union/European Economic Area (EU/EEA), newly adapted monovalent XBB.1.5 COVID-19 vaccines (XBB.1.5 vaccines) accounted for the majority of all known vaccines administered [14]. The SARS-CoV-2 XBB VSLs had been dominant in Europe from before the start of the vaccination campaign in 2023 until mid-December, when BA.2.86 VSLs (in particular JN.1) became predominant [15]. Since BA.2.86 VSLs were associated with potential immune escape [16], estimating the effectiveness of the XBB.1.5 vaccine against this new SARS-CoV-2 VSL was of utmost importance, together with assessing any difference in VE against XBB vs the subsequently predominant BA.2.86 VSLs. Some studies estimated VE against these time-consecutive SARS-CoV-2 VSLs separately using sequenced data, and compared estimates [4,5], while others directly compared the odds of vaccination between cases infected with the different VSLs, in case–case or case-only studies [17,18].

Here we propose two novel modelling approaches for estimating VE against time-consecutive VSLs (i.e., when there is a transition from one predominant VSL to another) when study-specific sequenced data is unavailable and proxy external data sources are used to identify predominantly circulating VSLs. To illustrate this, we used data from a European test-negative design study conducted in 2023/24 [11,19]. The proposed models permit a statistical comparison of the VE against two consecutive VSLs as well as VE estimation of each. In order to validate our proposed models, we compare the modelling results with those from the gold standard approach based on VSL cases classified using sequencing, and against the VE for each predominant period by using external data as a proxy.

Methods

Study design

The proposed modelling approaches were developed and applied in the context of test-negative design studies in the hospital setting to estimate VE. Very briefly, in this study design and setting [20], hospitalised patients are systematically recruited based on a clinical definition of severe acute respiratory infection (SARI) and undergo a laboratory test (preferably RT-PCR) for the target pathogen. Patients with an RT-PCR-positive result for SARS-CoV-2 are classified as “cases” and patients with a negative result as “controls”. Vaccine effectiveness is estimated as VE=(1-OR) x 100%, where OR is the estimated odds ratio of vaccination between cases and controls, adjusted for confounding, and obtained using logistic regression models.

Usual approaches for VSL-specific VE estimation

We consider the gold standard approach to estimate the VSL-specific VE as when VSL cases within the study are classified using sequenced data (“SD method”). We consider the VSL predominance period approach (“standard PP”) when VSL-specific VE is estimated using an external sequenced data source to classify VSL cases. In this approach, the study period is restricted to the period of predominance of the VSL of interest, i.e., all cases are considered to have been likely infected by a VSL of interest if the date of onset/swab falls within the period of that VSL predominance. Usually, the VSL predominance period is defined as the sequential weeks where the proportion of all sequenced viruses belonging to the VSL under study is above a predefined threshold (e.g., 60%), calculated using population-level surveillance data from an external, publicly available source (e.g., GISAID [21], European Respiratory Virus Surveillance Summary (ERVISS) [22]). For both of these methods, VE is calculated in the standard way as described in the previous section.

New modelling approaches

Our proposed novel modelling approaches for estimating VSL-specific VE use an “adapted PP” method, during the period of transition from the predominance of one VSL to another. For this purpose, instead of restricting the analysis period to each individual VSL predominance period determined from an external source and performing an analysis for each period, we introduce a new independent variable in the regression model to represent the VSL. This can be done in two ways: (1) as a categorical variable, that assumes the value of 0 for participants recruited during the period of predominance of the earlier VSL, and 1 for participants recruited during the period of predominance of the subsequent VSL (the “VSL category model”: VCM); and (2) as a continuous variable with values ranging from 0 to 1, representing the proportion of the subsequent VSL (the VSL gaining dominance) out of all sequenced viruses each week (the “VSL proportion model: VPM”). We summarise the four methods/approaches in Table 1.

Table 1: General description of the usual standard techniques and proposed novel model techniques to estimate variant/sublineage (VSL)-specific VE, VEBIS hospital study, Europe.

Test scenario.

To exemplify the described approaches to estimate VSL-specific VE, we used data from patients hospitalised with SARI from the Vaccine Effectiveness, Burden and Impact Studies (VEBIS) hospital network from September 2023 to April 2024, covering the period of transition between SARS-CoV-2 VSLs XBB to BA.2.86 predominance. Details of the methods used for standard COVID-19 VE analyses in the VEBIS hospital study are already published [11,19,23] as well as the results from this period [19].

Definitions used in current analyses (including of vaccinated and unvaccinated patients, and analytic exclusions), as well as a description of the proxy determination for the XBB and BA.2.86 predominance periods, can be found in S1 Table and S1 Section, respectively. We defined each study site-specific start study period as 14 days after the introduction of the XBB.1.5 vaccine within each country (S2 Table); there were also further restrictions depending on the modelling approach (see descriptions of each approach below). We estimated VE against each of the XBB and BA.2.86 VSLs as a “test scenario” using the VCM and the VPM modelling techniques. For “BA.2.86” we include all of its sublineages, such as JN.1. We validated our results by also estimating VE using both SD and the standard PP methods (when sample size allowed; see next section).

We restricted analysis to those vaccinated <60 days before symptom onset (where the period of the VSLs overlap) and to patients aged ≥65 years.

Ethical approval: The planning, conduct and reporting of the studies was in line with the Declaration of Helsinki. Official ethical approval was not required if studies were classified as being part of routine care/surveillance (Spain, Ireland, Malta); In Belgium and Germany, VE estimation is included in SARI surveillance. For Belgium, the study protocol was approved by the central Ethical Committee (CHU Saint-Pierre (AK/12-02-11/4111) initially in 2011, and UZ VUB (B.U.N. 143201215671) from 2014 on) and each participating hospital’s local ethical committees. The most recent amendment was approved on 27/9/2023 (reference 2012/310 Am6). The German SARI surveillance was approved by the Charité-Universitätsmedizin Berlin Ethical Board (Reference EA2/218/19). Other study sites obtained local ethical approval from a national review board (Croatia: 3 July 2023 by the Ethics committee of the Croatian Institute of Public Health, Class 030–02/23–01/3; Hungary: approved in March 2021 by the National Scientific and Ethical Committee for the period 01 September 2021–01 September 2024 (IV/1885–5/2021/EKU); Lithuania: approved 03 July 2020 by the Lithuanian Biomedical Research Ethics Committee No.: L-20–3/1–2; and later permission extended for the study period for seasons 2020–2024; Navarra: PI2020/45; Portugal: approved 19 January 2021 by the Ethics Committee of Instituto Nacional de Saúde Doutor Ricardo Jorge, no registration number given; Romania: approved by the Ethics Committee of the Ministerul Apărării Naionale Institutul Naional de Cercetare pentru Dezvoltare Medico-Militară, Cantacuzino” for the period 2022–2023, No. CE199/2022 and by the Ethics Committee of the National Institute of Public Health Romania No. 23468/2023).

Statistical analysis

For all approaches (SD, standard PP, VCM, and VPM), we first estimated the OR of vaccination between cases and controls with study site as a fixed effect and adjusted for confounding by date of symptom onset, sex, age and presence of at least one chronic condition (diabetes, heart disease, lung disease/asthma and immunodeficiency). The best functional forms of the continuous variables age and onset date (categories, splines, linear terms) were selected using the Akaike information criterion (AIC). We estimated VE overall (14–59 days) and by time since vaccination (TSV) in 30-day bands (14–29 and 30–59 days post vaccination). For the TSV analysis, vaccination was included as a categorical variable (unvaccinated, 14–29 days, 30–59 days post vaccination).

As described previously [11], we do not show estimates if there were no vaccinated cases, < 20 vaccinated patients or when the VE estimate had an absolute difference >10% from the VE estimated using penalised logistic regression (to assess small sample bias). We carried out a complete case analysis.

The SD (sequenced data) approach.

Here, cases were defined as those with a sequenced biological sample and classified as SARS-CoV-2 VSLs BA.2.86 or XBB. As the XBB period was earlier, to estimate XBB.1.5 VE against XBB VSL, we restricted the end of this study period for analysis as the date of onset of the last sequenced XBB case, at study site level. To estimate XBB.1.5 VE against BA.2.86 VSL, we defined the start of the study period as the date of onset of the first BA.2.86 sequenced case and the end as the date of onset of the last patient included at the time of the analysis. Since each study period was defined by the duration of the identification of the VSL of interest by sequencing, there could be an overlap in study periods and consequently, in each study site, some controls included in the models for XBB might also be included for BA.2.86.

PP (standard VSL predominance period) approach.

In the standard PP approach, we obtained XBB.1.5 VE against XBB and BA.2.86 VSLs by restricting the analysis to the periods of XBB and BA.2.86 predominance, respectively. Two models were fitted to the data restricted to each VSL predominance period to estimate each VSL VE. The definition of the study periods for XBB and BA.2.85 predominance is described in detail in S1 Section. Briefly, we used a threshold of 60% samples sequenced (using publicly available data from ERVISS 21]) to determine the weeks in each country during which each VSL was predominant (shown visually in Fig 1a and b, and with start dates listed in S3 Table).

Demonstration of the study period definitions for the predominance period approaches.(a) standard PP VE against XBB VSL, (b) standard PP VE against BA.2.86 and novel model approaches: (c) the VSL category (0 if XBB and 1 if BA.2.86) model and (d) VSL proportion (prevalence of BA.2.86 VSL) model. VEBIS hospital network, Europe. Standard PP: predominance period approach – standard technique using proxy for sequenced data; VCM: novel VSL category model; VEBIS: Vaccine Effectiveness, Burden and Impact Studies; VPM: novel VSL proportion model; VSL: variant/sublineage.

Adapted PP VCM (predominance period with VSL categorical variable) approach

In the proposed adapted PP with VCM, we included in the model a binary “VSL variable” to represent the most likely underlying VSL causing the infection (0 for XBB, 1 for BA.2.86 in this test scenario), and the interaction term between the VSL and vaccination variables (Eq. 1).

$$[eqn]$$

In this test scenario, the VSL variable represents XBB and BA.2.86 when the weekly BA.2.86 VSL proportion is ≤ 40% or ≥60%, respectively (Fig (c)). W represents the number of weeks of the study period, S, the number of sites and A the number of age groups, assuming time and age as discrete variables (the best functional form of these variables based on the AIC might differ across models).

Based on model Eq. 1 and for this test scenario, VE against the XBB VSL is calculated as $[eqn]$ , where the OR of vaccination against XBB ( $[eqn]$ ) is given by $[eqn]$ . Similarly, the VE against the BA.2.86 VSL can be calculated as $[eqn]$ , where the OR of vaccination against BA.2.86 ( $[eqn]$ ) is given by $[eqn]$ .

To statistically compare the VE against each VSL, we can calculate the ratio of the ORs (ROR), $[eqn]$ , which is given by the exponential of the interaction term in the model ( $[eqn]$ ), and its p-value. The ROR is a relative measure used to compare the OR from two groups, i.e., to assess if the relationship between an exposure of interest and an outcome differs between two groups. If the $[eqn]$ , then there are no differences in VE against XBB and BA.2.86 VSLs; if $[eqn]$ , the OR against BA.2.86 is higher than the OR against XBB VSLs (so the VE against BA.2.86 is lower than against XBB VSLs); and if $[eqn]$ , the VE against BA.2.86 is higher than against XBB VSLs.

It can be also proven that the OR of vaccination estimated by case–case modelling approaches [24], when controls are the same, corresponds to the ratio of the OR between the two VSLs ( $[eqn]$ ) (S3 Section).

For the models used to estimate VSL VE by TSV, deductions can be made in a similar way, noting that now that the vaccination variable has more than two categories.

Adapted PP VPM (VSL as a continuous variable) approach.

In the proposed adapted PP with VPM approach, the VSL variable in Eq. 1 is represented as a continuous numeric variable with values ranging from zero to one, depicting the population level relative frequency of SARS-CoV-2 cases that belong to the VSL BA.2.86. This modelling approach allows the estimation of the VSL-specific VE at any value of the BA.2.86 VSL proportion (Fig 1d). In Eq. 1, if we assume VSL as the proportion of cases classified as BA.2.86 VSL, the OR of being vaccinated in cases vs controls can be written as a function of the BA.2.86 VSL proportion of interest $[eqn]$ , as presented below:

$$[eqn]$$

When $[eqn]$ , VE is being estimated during weeks when the prevalence of BA.2.86 is null, if all the SARS-CoV-2 cases belonged to XBB VSL, $[eqn]$ . Likewise, when =1, VE is being estimated during weeks when the proportion of BA.2.86 is 1, so $[eqn]$ .

For both the VCM and VPM approaches, the BA.2.86 VSL proportion weekly time series at study level was smoothed by fitting a logistic regression to the weekly proportion of BA.2.86 VSL extracted from ERVISS (S2 Section and S1 Fig). If the study site weekly BA.2.86 proportion was missing, the values were imputed using data from neighbouring countries (to overcome weeks with missing data and/or with a low number of sequenced samples).

As a sensitivity analysis, to enable comparison of results from the VPM with the other approaches, we estimated the OR when VSL = 0.3 for the VE against XBB and VSL = 0.7 for the VE against BA.2.86. More specifically $[eqn]$ and $[eqn]$ .

Results

Within our “test scenario” period (September 2023–April 2024), the start date for data included in both SD and standard PP XBB analyses was 02 October 2023, while the BA.2.86 start dates were 09 October 2023 (for SD analyses) and 22 November 2023 (for standard PP analysis). End dates were in December 2023 for XBB analyses (07 December for PP and 19 December for SD analyses) and in April 2024 for BA.2.86 (25 April for SD and 26 April for standard PP). For both VCM and VPM approaches, the study period was from 02 October 2023 to 26 April 2024. The final study periods, number of SARI patients included and number and sites included in the standard technique and novel model analyses are shown in Table 2.

Table 2: Final study periods, numbers of SARI patients and number and sites included in analyses using standard techniques and proposed novel model techniques, VEBIS hospital study, Europe.

Vaccine effectiveness results

Overall, in the first 14–59 days post vaccination, as there were no vaccinated cases in the SD analysis for VE against XBB VSLs, we could only validate the models against the standard PP (Table 3).

Table 3: Effectiveness of the monovalent XBB.1.5 COVID-19 vaccines against hospitalisation among individuals aged ≥65 years olda, by VSL (XBB or BA.2.86) and by time since vaccination (TSV), using four modelling approaches: sequence data (SD), standard predominance period approach (PP), adapted predominance period with VSL categorical variable approach (VCM) and with VSL proportion variable approach (VPM). VEBIS hospital network, Europe, 2023/24.

The VE against XBB VSLs for this period was 68% (95%CI: 38–86) using the VCM, 71% (95%CI: 50–83) with the VPM and, in validation, 67% (95%CI: 1–92) using standard PP (Table 3). Against BA.2.86 VSLs, VE was 41% (95%CI: 23–54) using the VCM, 48% (95%CI: 35–59) with the VPM and, for validation, VE was 34% (95%CI: −14–64) in the SD analysis and 42% (95%CI: 25–56) using standard PP.

At 14–29 days post vaccination, due to small sample sizes, the VE using SD was not possible for either VSL and the models could only be validated using standard PP. Against XBB VSL, VE was 75% (95%CI: 45–90) using the VCM, and 77% (95%CI: 52–89) using the VPM, while the standard PP validation VE was 88% (95%CI: 43–99). Against BA.2.86 VSL, VE was 36% (95%CI: −8–62) for the VCM, 43% (95%CI: 13–63) using the VPM, and 38% (95%CI: −2–64) in validation using standard PP.

At 30–59 days post vaccination, low sample size precluded any estimates using the VCM. The VE against XBB VSLs was 62% (95%CI: 16–83) using the VPM. Against BA.2.86 VSL, VE was 42% (95%CI: 23–56) using the VCM, and 46% (95%CI: 30–59) using the VPM. Validation using SD was 39% (95%CI: −12–69) and 43% (95%CI: 25–57) using standard PP.

The results from the proposed methods, and well as from the standard techniques, show a decline in VE as BA.2.86 VSLs increased in circulation, overall and by TSV (Fig 2).

Effectiveness of the monovalent XBB.1.5 COVID-19 vaccines against hospitalisation among individuals aged ≥65 years old, by BA.2.86 VSL proportion and by time since vaccination, using the four approaches: the gold standard sequencing data (gold standard SD), standard predominance period (standard PP), adapted predominance period with VSL category model (PP VCM) and with the proportion model (PP VPM); VEBIS hospital network, Europe.

Comparison between VE against XBB and BA.2.86 VSLs

The ROR against BA.2.86 and XBB VSLs in the first 14–59 days post vaccination was 1.9 (95%CI: 0.9–4.3) and 1.8 (95%CI: 1.1–2.9) using the VCM and the VPM, respectively (Table 4). At 14–29 days post vaccination, the ROR against BA.2.86 vs XBB VSLs was 2.6 (95%CI: 1.0–7.6) with the VCM and 2.4 (95%CI: 1.2–5.0) with the VPM. At 30–59 days post vaccination, the ROR against BA.2.86 vs XBB VSLs was 1.4 (95%CI: 0.7–2.7) using the VPM (small sample size precluding results using the VCM).

Table 4: Comparison of the effectivenessa of the monovalent XBB.1.5 COVID-19 vaccines against XBB and BA.2.86 VSLs among hospitalised SARI patients aged ≥65 years, by TSV, using the adapted predominance period (PP) with VSL category model (VCM) and with VSL proportion model (VPM) approaches, VEBIS hospital network, Europe.

The estimation of the ratio of ORs ( $[eqn]$ ) with the BA.2.86 variant proportion shows an increased risk of hospitalisation among those who were vaccinated and infected with BA.2.86 vs those vaccinated and infected with XBB VSLs, being statistically significant overall (14–59 days) and at 14–29 days post vaccination (Fig 3).

Comparison of the effectivenessa of the monovalent XBB.1.5 COVID-19 vaccines against XBB and BA.2.86 VSLs among hospitalised SARI patients aged ≥65 years, by BA.2.86 VSLs using adapted PP with VPMb, by TSV, VEBIS hospital network, Europe.OR: odds ratio; PP: standard predominance period; TSV: time since vaccination; VE: vaccine effectiveness; VEBIS: Vaccine Effectiveness, Burden and Impact Studies; VPM: VSL proportion model; VSL: variant/sublineage. a The comparison of the VE is measured by the ratio of the OR of vaccination between cases and controls against BA.2.86 and XBB VSLs (ORBA.2.86/ORXBB), for the proportion of circulating BA.2.86 VSLs ≥ 60%. The ratio of OR is then calculated as ORBA.2.86/ORXBB=ORp/OR1−p, where p is the proportion of BA.2.86 VSLs. b The VPM model is a logistic regression model having case/control status as the dependent variable and the date of symptom onset, age, sex, presence of a chronic condition, vaccination status, VSL proportion and an interaction term between vaccination and the VSL proportion as independent variables. The form used to include date of symptom onset (linear, week, month, splines) and age (10-year age group, linear, splines) in the model was selected as the model with the lowest AIC.

Discussion

We propose two different modelling approaches as an adaptation of the standard PP approach (VCM and the VPM), that allow for the estimation of VSL-specific VE as well as for statistical comparison of the VE against two different VSLs, when linked sequencing data are not available and inferring lineage through PCR techniques is not possible. We compared the VSL-specific VE results from our proposed models using the gold standard technique using linked SD and by using a proxy restricting the study period to periods of predominant circulation of the VSLs of interest (standard PP). Both validation techniques are well-established methodological approaches [1,4–11].

Results from our proposed modelling approaches produced similar VE estimates up to 59 days post vaccination (≤7% absolute difference) to those produced using established methods, varying from 68% to 71% for VE against XBB VSLs and from 41% to 48% for VE against BA.2.86 VSLs, using the VCM and with the VPM, respectively. Similar results from the different approaches were also found using narrower TSV intervals. Our study results also suggest that VE was lower against BA.2.86 than against XBB VSL hospitalisation, with estimates of the ROR of vaccination against BA.2.86 vs that against XBB ranging between 1.8 and 1.9 for the two proposed modelling approaches.

Other European studies and two US studies have found similar VSL comparative VE results; all suggested a lower VE against BA.2.86 than XBB VSLs [4,5,7,12], albeit with some differences in study design and population. Results from case-only studies also suggest lower VE against BA.2.86 than against XBB VSLs [17,18]. Huiberts et al. found vaccinated participants to have higher odds of BA.2.86 vs XBB infection, of 1.5 (95%CI: 0.8–2.6) [17], while Moustsen-Helms et al. estimated an OR of 1.5 (1.3–1.9) for BA.2.86 vs a non-BA.2.86 variant and 1.6 (1.3–2.0) for JN.1 lineage vs a non-BA.2.86 variant [18]. Estimates are slightly lower than ours; however, differences in study population, TSV, study period and study design (these studies report VE against infection, not hospitalisation) might all contribute to the differences found.

Both proposed models have many strengths: (1) they allow for the estimation of VE against each VSL in the same model, increasing the sample size used and consequently the precision, compared with the standard PP approach and VE using sequencing results (both of which use separate models for each study period). (2) The VPM, in our study, was able to provide VE estimates when standard PP and SD approaches were not able to do so due to low sample size. (3) The models also allow for a statistical comparison of VE against different VSLs, providing an estimate of a relative measure (the ratio of the OR of vaccination against two consecutively circulating VSLs), directly comparable with the OR reported from case–case studies. (4) Neither proposed model requires the use of linked sequencing and epidemiological data and the VPM does not require a threshold definition to determine the predominant VSL, which can be a limitation of the other techniques. (5) Finally, the VPM also allows the estimation of both indicators (VE and the ratio of the OR of vaccination against different VSLs) in a continuous form, providing more information about the measures of interest.

The proposed models also have limitations: (1) although the proposed methodology aims to overcome the lack of linked sequencing data to investigate VE differences against different VSLs, it still relies on SARS-CoV-2 sequencing data from surveillance made publicly available by countries, using a proxy for the VSL causing the infection at an ecologic level, which can introduce bias, reduce precision, and may not be representative of the study population. Additionally, two study sites used information from neighbouring countries, though this accounted for only a small proportion, less than 9%, of the study population. The magnitude and direction of the bias will depend on the magnitude and direction of the measurement error of the VSL proportion but also on the true difference in VE between the two VSLs. In this test scenario, VE estimates from our VCM and VPM models produced similar estimates as the models using sequencing data (when available) and to other studies also using sequencing data [4,5]. We were also able to identify differences in VE between VSLs. These results may signal that if there was a measurement error bias, it was low. Additionally, both data sources, the study population and the population from which the VSL data is estimated, are originated from surveillance systems, so they should be more closely related. (2) Using an external data source also requires having adequate sample size for robust measures of the VSL proportions. (3) These models are only useful with time-consecutive VSLs transitioning from predominance of one to the other. This method might not therefore be transferable to other pathogens not experiencing such transitions. For example, in seasonal influenza subclade VE estimation, subclades may be co-circulating rather than transitioning from dominance of one to another, rendering this method unfeasible. However, it would be interesting to test these models on VE against time-sequential transitioning seasonal influenza (sub)types, which can occur. It would also be useful to test these models on VE in other settings. Future work exploring the full capacity and limitations of these models are essential for their broader applicability.

As in all observational studies, limitations also include the potential residual confounding that can remain even after adjustments (e.g., from vaccinated population differences over time). In addition, despite stratification by TSV in narrow 30-day time bands, adjustments for time of symptom onset and potential confounders, the inclusion of the underlying circulating VSL (included either as a categorical variable or as a proportion), and the short study period, it is unclear if we can fully disentangle the effect from TSV and the circulating VSL. This challenge is illustrated by the VPM results. We observe that precision of VE as a function of the BA.2.86 proportion decreases when the proportion of BA.2.86 increases at 14–29 days post-vaccination, and also decreases when the proportion of BA.2.86 is lower, at 30–59 days post-vaccination. However, for the overall analysis, the median TSV for the BA.2.86 VE estimate is higher than for the XBB estimate (45 vs 20–21 days), which could explain the differences in VE, despite the short overall period post vaccination. Using narrower TSV bands allows us to see a significant effect at 14–29 days, when median TSV is similar for both VSL estimates (varying between 19 and 23 days).

Further limitations were that we were unable to include all 12 sites in the analysis, and that there were different sites included in each of the validation approaches, which could lead to heterogeneity and lack of comparability of the analysis. However, the same sites were included for both VCM and VPM approaches, and results were similar between the proposed models and the standard techniques. In addition, the controls included in each approach may differ. For the SD, PP and VCM approaches, patient inclusion is performed by restricting patients over time, which means that some cases and controls recruited for each method may be different. In the SD approach, there might be some overlap of time and also of controls included for each of the XBB and BA.2.86 models. For both the standard PP and VCM approaches, however, controls do not overlap between VSL models by study site. while all cases and controls across both VSL periods are included in the VPM approach. Further analysis is required to investigate the impact of these changes in included sites and/or controls for each approach.

Small sample size precluded some results. In particular, it was not possible to estimate VE against XBB VSLs using SD due to no vaccinated cases, which would have been very relevant to validate our VE estimates against XBB. However, results from other studies have reported similar estimates [4], or report higher VE against XBB VSLs than against BA.2.86 (or JN.1) VSLs, despite differences in study design [5].

In conclusion, our two proposed novel modelling approaches provided similar VE estimates with generally improved precision compared with well-established methods. In particular, both VCM and VPM had greater precision than when using SD in our study, as we had fewer linked samples. The VPM was also able to provide VE estimates when both standard PP and SD techniques could not, due to low sample size. Although the optimal estimates would ideally integrate sequencing with epidemiological data, we anticipate that our novel modelling techniques will encourage other researchers lacking extensive study-specific sequencing data to provide more robust VE estimates.

Supporting information

S1 TableDefinitions used for analyses, including analytic exclusions, VEBIS hospital study, Europe.(PDF)

S1 SectionDescription of the proxy determination for XBB and BA.2.86 predominance periods.(PDF)

S2 SectionDescription of the estimation of the weekly BA.2.86 VSL proportion.(PDF)

S1 FigWeekly BA.2.86 VSL proportion reported by the country, the proportion calculated using neighbouring countries and the final estimated time series using logistic regression used in the category and proportion models, based on TESSy/GISAID data (ECDC ERVISS Github), VEBIS hospital network, Europe.(PDF)

S2 TableStart date of the 2023/24 vaccination campaign by site, VEBIS hospital study, Europe.(PDF)

S3 TableStart date^a^ of the XBB and BA.2.86 VSL predominant periods for a 60% predominance threshold,^b^ by site, VEBIS hospital study, Europe.(PDF)

S3 SectionDemonstration of ratio of odds ratios (ORs) in case–case studies.(PDF)

S2 FigPatient exclusion flowchart for the XBB sequenced data (SD) analysis, VEBIS hospital study, Europe.(PDF)

S3 FigPatient exclusion flowchart for the BA.2.86 sequenced data (SD) analysis, VEBIS hospital study, Europe.(PDF)

S4 FigPatient exclusion flowchart for the XBB standard predominance period (standard PP) analysis, VEBIS hospital study, Europe.(PDF)

S5 FigPatient exclusion flowchart for the BA.2.86 standard predominance period (standard PP) analysis, VEBIS hospital study, Europe.(PDF)

S6 FigPatient exclusion flowchart for the XBB and BA.2.86 adapted predominance period (adapted PP) variant/sublineage category model (VCM) analysis, VEBIS hospital study, Europe.(PDF)

S7 FigPatient exclusion flowchart for the XBB and BA.2.86 adapted predominance period (adapted PP) variant/sublineage proportion model (VPM) analysis, VEBIS hospital study, Europe.(PDF)