Evaluating the accuracy of survey data: a case study of COVID-19 vaccination rates in Germany

TL;DR

This paper evaluates how accurate survey data on COVID-19 vaccination rates in Germany are compared to official records, showing that survey design and weighting methods affect accuracy.

Contribution

The study provides empirical evidence on the accuracy of survey estimates of vaccination rates and the impact of survey design and weighting techniques.

Findings

Early surveys underestimated vaccination rates, while later ones overestimated them.

Probability-based mixed-mode or personal interview surveys were more accurate than other designs.

Adjustment weights generally improved the accuracy of survey estimates.

Abstract

Surveys are an important source of timely and comprehensive population health data and play a crucial role in public health research and policymaking, as shown during the COVID-19 pandemic. However, the reliability of survey data depends on their accuracy, which is often difficult to assess due to the limited availability of benchmark data. This study evaluates the accuracy of survey estimates of the COVID-19 vaccination rate in Germany and examines the impact of survey design and adjustment weights on accuracy.

I compared survey estimates of the COVID-19 vaccination rate from multiple surveys conducted in Germany between 2021 and 2022 against administrative data on the vaccination rate as an external benchmark. Accuracy was assessed by calculating absolute and relative deviations between survey estimates and the administrative data. Further, I analyzed accuracy differences based on survey design, focusing on sampling procedure and survey mode, and I examined whether adjustment weights improved estimation accuracy.

The accuracy of survey estimates varied over time, with early surveys underestimating the vaccination rate and later surveys tending to overestimate it. Probability-based mixed-mode or personal interview surveys yielded more accurate estimates than did other survey designs. While some nonprobability web surveys performed well, their accuracy varied considerably. The application of adjustment weights generally improved estimation accuracy, suggesting that this weighting technique effectively addressed some sources of survey error.

Survey-based estimates of COVID-19 vaccination rates should be interpreted with caution due to widespread inaccuracies, particularly overestimation. Probability mixed-mode and personal interview surveys produced more accurate estimates, underscoring the importance of robust survey methodologies. The fact that I found that adjustment weights enhanced accuracy highlights their value in survey research. These findings provide valuable insights for public health researchers and survey methodologists in designing and interpreting health-related survey data.

The online version contains supplementary material available at 10.1186/s12874-025-02702-2.

Figures

Timeline of the COVID-19 vaccination campaign in Germany: Vaccination rate (at least 1 dose) and selected policiesData sources: Vaccination rates – Robert Koch Institute (RKI) [[38](#CR38)] and German Federal Statistical Office (Destatis) [[39](#CR39)]; vaccine availability and policies – German Federal Ministry of Health chronicle [[36](#CR36)] and the Oxford COVID-19 Government Response Tracker [[40](#CR40)]

Absolute accuracy of the survey estimates of the COVID-19 vaccination rate with the mean benchmark value over the fieldwork period, age group 18–59 years

Absolute accuracy of the survey estimates of the COVID-19 vaccination rate compared with the mean benchmark value over the fieldwork period, age group 60 years and older

Absolute accuracy of estimates of the COVID-19 vaccination rate by survey design group compared with the mean benchmark value over the fieldwork period, age group 18–59 years

Absolute accuracy of the estimates of the COVID-19 vaccination rate by survey design group compared with the mean benchmark value over the fieldwork period, age group 60 years and older

Absolute accuracy of survey estimates of the COVID-19 vaccination rate with and without adjustment weighting compared with the mean benchmark value over the fieldwork period, age group 18–59 years

Absolute accuracy of survey estimates of the COVID-19 vaccination rate with and without adjustment weighting compared with the mean benchmark value over the fieldwork period, age group 60 years and older

Full text

Background

Surveys have proved to be an important source of population information on health status, health determinants, and access to healthcare. The COVID-19 pandemic underscored their critical role, as surveys enabled researchers to monitor the rapidly evolving health situation and supported policy makers in combating the crisis. Examples of such studies include the COVID-19 Household Pulse Survey in the United States [1], the Household Impacts of COVID-19 Survey in Australia [2], the Coronavirus Infection Survey in the United Kingdom [3], and the COVID-19 Snapshot Monitoring in Germany [4].

The accuracy of survey data is a prerequisite for reliable and unbiased findings, but its empirical evaluation is challenged by the lack of benchmark data for comparison. Accuracy assessments focus mostly on comparing distributions of sociodemographic variables in the samples with external benchmarks provided by official statistics [5–7] and on comparing self-reported voting behavior with the actual election outcome as a benchmark [8–10]. Together, these studies show that accuracy varies largely across surveys.

According to the well-established total survey error (TSE) framework, accuracy of survey data can be improved through different actions at each stage of the survey process [11–13]. At the design stage, the choice of two key survey design characteristics – sampling procedure and survey mode – has a decisive impact on survey accuracy. Regarding sampling procedure, multiple studies have concluded that probability surveys are more accurate than nonprobability surveys [6–9, 14, 15], a result reached also in a meta-analysis by Cornesse and Bosnjak [16]. Regarding survey mode, the evaluation of its impact on accuracy is complex because it influences various types of survey error. Mode-specific errors include coverage error, such as the exclusion of certain population groups (e.g., the offline population in web-based surveys [17–19]), and measurement error, such as social desirability bias in sensitive questions (particularly in interviewer-administered surveys [20–22]. Despite the extensive literature examining individual error sources, only few studies have evaluated the overall impact of survey mode on survey accuracy [23, 24].

Moving to the stage of post-survey adjustments, survey weights can potentially improve survey accuracy. However, findings on the impact of adjustment weights on accuracy are mixed. Whereas some studies have found that adjustment weights improved accuracy [25–28], others have observed little difference in or a deterioration of accuracy [5, 6, 10, 14]. Furthermore, some studies have shown that the impact of adjustment weights differs across surveys, outcome variables, and the methods of comparison used for the assessment of accuracy [7, 15, 29, 30].

The provision of accurate information is a general requirement for public health studies, but the COVID-19 pandemic introduced additional challenges for survey research. First, the goal of maximizing accuracy was constrained by the additional requirement for timely delivery of results. As multiple studies have indicated, these parallel requirements pose the danger of trading established survey procedures for those with lower accuracy but higher speed [31–34]. In addition, contact restrictions limited the feasibility of the traditional interviewer-administered surveys and supported the use of self-administered survey modes. Assessing the accuracy of studies conducted during the pandemic — and the link between survey design and accuracy— is therefore of particular importance. For this purpose, I use data on a health-related substantive measure of high policy relevance, the COVID-19 vaccination rate. Focusing on surveys conducted between 2021 and 2022 in Germany, I investigate the following research questions:

• RQ1. How accurate were the survey estimates of the COVID-19 vaccination rate in Germany?
• RQ2. Did the accuracy of the survey estimates of the COVID-19 vaccination rate differ by survey design?
• RQ3. Did the application of adjustment weights improve the accuracy of the survey estimates of the COVID-19 vaccination rate?

By addressing these questions, I seek to provide a comprehensive evaluation of survey accuracy under the exceptional conditions of the COVID-19 pandemic and thereby offering valuable insights for both public health researchers and survey methodologists. In the following, I first provide contextual information on the COVID-19 vaccination campaign in Germany. Next, I introduce the data sources, measures, and methods used in this study. I then present the empirical results, discuss them in relation to existing research, and acknowledge the study’s limitations. I conclude with a summary and offer recommendations derived from the study.

COVID-19 vaccination campaign in Germany

In Germany, COVID-19 vaccinations played an important role in the fight against the spread of the virus. Figure 1 maps some of the important milestones in the vaccination campaign in Germany between December 2020 and August 2021. The COVID-19 vaccination campaign began in Germany in late December 2020. The first vaccines were administered to the priority groups, including the elderly, those with high-risk health conditions, residents and staff of long-term care facilities, and selected occupational groups with increased exposure risk [35]. Eligibility was defined not only by residence status but also included individuals insured under the German health system or employed in Germany. Over the following months, the vaccines became available for younger age groups in the general population, as the events marked in red in Fig. 1 show. In June 2021, the prioritization order was lifted. From this point forward, all adults aged 18 or over living, working, or insured in Germany, regardless of citizenship, became eligible for COVID-19 vaccination [36, 37].Fig. 1. Timeline of the COVID-19 vaccination campaign in Germany: Vaccination rate (at least 1 dose) and selected policiesData sources: Vaccination rates – Robert Koch Institute (RKI) [38] and German Federal Statistical Office (Destatis) [39]; vaccine availability and policies – German Federal Ministry of Health chronicle [36] and the Oxford COVID-19 Government Response Tracker [40]

The development of the vaccination rate (gray lines in Fig. 1) reflects the accessibility of the vaccine. For both depicted age groups, the proportion of those vaccinated at least once grew until August 2021. During this time, the importance of vaccination to mitigate the spread of the virus was continuously emphasized by policymakers and encouraged by several political actions. Two examples of such actions introduced in May 2021 – the easing of restrictions for vaccinated persons and the introduction of advantages for vaccinated persons when entering Germany – are marked in blue in Fig. 1. Further policies and vaccination efforts followed in autumn and winter of that year, as the pace of the increase in the vaccination rate started to slow down and another pandemic wave arose [36, 41].

Data and methods

Data

The analysis is based on two data types – survey data and official health statistics. For the identification of relevant surveys, I relied on the SDCCP 1 dataset, a systematic collection of academic quantitative surveys conducted in Germany between March 2020 and December 2021 [42, 43]. This dataset covers a total of 717 surveys clustered within 183 survey programs. For the present study, I systematically screened the dataset for surveys that included a question on the respondent’s COVID-19 vaccination status. Furthermore, I included only surveys that covered the general resident population in Germany (based on the defined target population), excluding those aimed at narrower subgroups. The search within the SDCCP 1 dataset and the subsequent request for datasets resulted in 50 eligible surveys with available data clustered within 18 survey programs (e.g., rounds or waves).

To acknowledge the considerable differences in vaccine availability – and thus in the vaccination rate – by age, I divided the analysis into two age groups – 18–59 years and 60 years and older. The survey vaccination rate was calculated for each age group. However, some surveys did not allow a division into these two age groups, as either the age span of the survey target population differed from the required span, or information on respondents’ exact ages was not available. Thus, the final analytical samples for RQ1 and RQ2 comprised 34 surveys for the age group 18–59 years and 50 surveys for the age group 60 years and older. For the analyses for RQ3, I included only probability and non-probability surveys that provided adjustment weights in the datasets, allowing for a comparison of estimates with and without adjustment weights. Consequently, the analytical datasets for RQ3 were further reduced to 23 surveys in the age group 18–59 years and 39 surveys in the age group 60 years and older.

The official health statistics that served as benchmark data for my study were obtained from the Robert Koch Institute (RKI), a German public health institute that collected administrative data during the COVID-19 pandemic. Specifically, the benchmark data employed in the analyses included the number of vaccines administered daily [38]. To calculate the benchmark vaccination rate, I combined the benchmark data with population figures provided by the German Federal Statistical Office (Destatis) [39].

The analyses presented in this study are based on a customized dataset specifically created for this purpose. For each included survey, the dataset includes the survey estimate of the vaccination rate, variables capturing survey design features (as provided in the SDCCP 1 dataset), and the corresponding merged benchmark vaccination rate.

Measures and methods

The measure of interest in my analyses is the COVID-19 vaccination rate defined as the percentage of the population that had received at least one dose of the vaccine. In the surveys, the vaccination rate represents the percentage of respondents who reported receiving at least one dose of the vaccine and refers to the respective fieldwork periods. In the benchmark data, I calculated the vaccination rate by dividing the number of first vaccine doses administered by the population number. As the benchmark vaccination rate was available on a daily level, I decided on a match based on the mean benchmark vaccination rate over the respective fieldwork period. Comparing both values, I calculated the absolute accuracy as the percentage point difference in the vaccination rate between the survey estimate and the benchmark value. With this definition, positive values refer to the overestimation and negative values to the underestimation of the vaccination rate by surveys. I note that official health statistics provide the best available benchmark for vaccination rates but may still contain errors. They should therefore not be regarded as perfectly accurate representations of the population parameter. This issue is further addressed in the Discussion.

In addition, I conducted two robustness checks (RCs) to test the sensitivity of the absolute accuracy measure matched with the mean benchmark vaccination rate. First, instead of the mean benchmark vaccination rate over the fieldwork period, I used the benchmark vaccination rate on the first day of fieldwork (RC1). My rationale for this approach was that the proportion of survey interviews completed at the beginning of the fieldwork period might have been high compared with the days and weeks that followed [44, 45]. Second, instead of absolute accuracy, I calculated relative accuracy in percent, defined as the difference between the survey estimate and the benchmark divided by the survey estimate (RC2). With this measure, I aimed to take into account the difference in vaccination rate levels over time. For instance, a small absolute inaccuracy of a few percentage points at the beginning of the vaccination campaign might have translated into a comparatively high relative inaccuracy in percent if the overall – then low – levels of the vaccination rate had been considered.

For the analysis of the association between accuracy and survey design (RQ2), I evaluated two key survey design features – sampling procedure and survey mode. The combinations of the two features in my sample yielded six survey design groups: (1) probability surveys conducted via computer-assisted personal interviewing (CAPI); (2) probability surveys conducted via computer-assisted telephone interviewing (CATI); (3) probability surveys conducted via computer-assisted web interviewing (CAWI); (4) probability mixed-mode surveys conducted via paper-and-pencil interviewing (PAPI) and CAWI; (5) probability mixed-mode surveys conducted via CATI and CAWI; and (6) nonprobability surveys conducted via CAWI. Due to the low number of cases, I combined both types of probability mixed-mode surveys into a single group for the interpretation of the results.

For the analysis of the association between accuracy and survey weights (RQ3), I compared the accuracy of survey estimates with and without adjustment weights. In both cases, design weights were applied when available for probability surveys. For the analyses addressing RQ1 and RQ2, I applied all weights provided in the datasets (design and adjustment).

Results

RQ1 – Overall accuracy of survey estimates

Figure 2 shows the absolute accuracy of the survey estimates of the COVID-19 vaccination rate for the age group 18–59 years compared with the mean benchmark value over the fieldwork period. The accuracy varied largely across surveys, ranging from − 10.7% points (the largest underestimation) to + 17.8% points (the largest overestimation). For 29 of the 34 surveys (85%), the confidence intervals did not overlap with zero. This indicates that most surveys exhibited a statistically significant deviation from the benchmark. Specifically, 24 of the 29 surveys that exhibited a statistically significant deviation from the benchmark (83%) overestimated the vaccination rate, and five (17%) underestimated it. The median absolute inaccuracy among all surveys was 9.8% points.Fig. 2. Absolute accuracy of the survey estimates of the COVID-19 vaccination rate with the mean benchmark value over the fieldwork period, age group 18–59 years

The distribution further revealed that two groups could be distinguished: early surveys (n = 8), whose fieldwork began between February and June 2021, and later surveys (n = 26), whose fieldwork began in July 2021 or later. Whereas the vaccination rate estimates in the early surveys were relatively more accurate (absolute median = 2.6% points), and seven of the eight surveys in that group underestimated the vaccination rate, the estimates in the later surveys were less accurate (absolute median = 10.8% points), and 23 of the 26 surveys in that group (88%) overestimated the vaccination rate.

In addition to the main analysis, I conducted two robustness checks to validate the results on the accuracy of survey estimates in the age group 18–59 years. For robustness checks RC1 (Fig. S1 in the supplementary material) and RC2 (Fig. S2 in the supplementary material), the comparison of the absolute median accuracy confirmed the conclusion that early surveys were, on average, more accurate than later surveys.

Figure 3 shows the absolute accuracy of the survey estimates for the age group 60 years and older with the mean benchmark value over the fieldwork period. The magnitude of the range of inaccuracy (− 19.7 to + 9.8% points) was similar to that in the age group 18–59 years, but with greater negative deviations and smaller positive deviations. The median absolute inaccuracy was 6.2% points. For 37 of the 50 surveys included in the analysis (74%), the confidence intervals did not overlap with zero. This indicates that – similarly to the age group 18–59 years – most surveys exhibited a statistically significant deviation from the benchmark. Specifically, of the 37 surveys that exhibited a statistically significant deviation from the benchmark, 29 (78%) overestimated the vaccination rate and eight underestimated it.Fig. 3. Absolute accuracy of the survey estimates of the COVID-19 vaccination rate compared with the mean benchmark value over the fieldwork period, age group 60 years and older

Here, too, two groups could be distinguished: early surveys (n = 10), whose fieldwork began between February and May 2021, and later surveys (n = 40), whose fieldwork began in June 2021 or later. The pattern for the difference between early and later surveys observed in the age group 18–59 years was repeated in the older age group. While all 10 early surveys underestimated the vaccination rate, 29 of the 40 later surveys (73%) overestimated it. In terms of absolute accuracy, early surveys performed slightly better than later surveys (absolute median = 5.2 and 6.3% points, respectively).

To validate these results, I conducted the same robustness checks as in the age group 18–59 years. RC1 (Fig. S3 in the supplementary material) confirmed that the vaccination rate estimates in the early surveys were, on average, more accurate than in later surveys. However, RC2 showed the opposite pattern, with later surveys being more accurate than early surveys. Figure S4 in the supplementary material illustrates that the relative accuracy of the vaccination rate estimates deteriorated especially for the earliest surveys conducted in February and March 2021, a period with comparatively low levels of vaccination.

Overall, I conclude that surveys conducted during the first months of the vaccination campaign mostly underestimated the vaccination rate, whereas surveys conducted during the later phase tended to overestimate it. Furthermore, most of the accuracy measures applied showed that early surveys were more accurate than later ones. However, one of the robustness checks in the age group 60 years and older suggested that the results may depend on the measure applied, as they differed between the standard and relative accuracy measures during months with low vaccination rate levels.

RQ2 – Association between estimate accuracy and survey design

Figure 4 depicts the accuracy of the estimates of the COVID-19 vaccination rate by survey design in the age group 18–59 years. Probability mixed-mode surveys (PAPI & CAWI; CATI & CAWI) achieved the highest absolute accuracy (absolute mean and absolute median = 4% points), followed by the small group of two probability CAPI surveys (absolute mean and absolute median = 7% points). Whereas these two groups and the nonprobability CAWI surveys group displayed both under- and overestimation, all probability CAWI and probability CATI surveys overestimated the vaccination rate. In terms of variability, nonprobability CAWI surveys demonstrated the highest standard deviation (5.7% points). The robustness checks provided similar results in terms of accuracy (see Fig. S5 and Fig. S6 in the supplementary material). Overall, I therefore conclude that, of all survey designs considered in the age group 18–59 years, probability mixed-mode (PAPI & CAWI; CATI & CAWI) surveys and probability CAPI surveys achieved the highest accuracy.Fig. 4. Absolute accuracy of estimates of the COVID-19 vaccination rate by survey design group compared with the mean benchmark value over the fieldwork period, age group 18–59 years

Figure 5 depicts the absolute accuracy of estimates of the COVID-19 vaccination rate by survey design in the age group 60 years and older. Altogether, the differences in accuracy (measured by absolute mean and absolute median) across survey design groups were comparatively small, with a 4.5%-point gap between the highest and lowest absolute means and a 6.2%-point gap between the highest and lowest absolute medians. As in the case of the age group 18–59 years, the most accurate survey design group in terms of mean accuracy was the probability CAPI surveys group (only two observations, absolute mean and absolute median = 5% points) and the most accurate survey design group in terms of median accuracy was the probability mixed-mode (PAPI & CAWI; CATI & CAWI) surveys group (absolute mean = 7% points, absolute median = 3% points). However, probability mixed-mode surveys were also the survey design group with the highest standard deviation (8% points) and included two cases with some of the highest inaccuracies across all surveys. A possible reason for the relatively large underestimation of the vaccination rate in these two cases may be that the fieldwork took place during the early phase of the vaccination campaign. Among the other survey design groups, overestimation prevailed across probability surveys. Similarly to the age group 18–59 years, nonprobability CAWI surveys displayed high variability, including some of the most accurate survey estimates, as well as comparatively large under- and overestimates. The robustness checks confirmed these findings (Fig. S7 and Fig. S8 in the supplementary material). Overall, the conclusion from the analysis of the age group 18–59 years that probability CAPI surveys and probability mixed-mode (PAPI & CAWI; CATI & CAWI) surveys were the most accurate design groups holds also for the age group 60 years and older, although the differences across survey design groups were relatively small.Fig. 5. Absolute accuracy of the estimates of the COVID-19 vaccination rate by survey design group compared with the mean benchmark value over the fieldwork period, age group 60 years and older

RQ3 – Association between estimate accuracy and adjustment weights

For the evaluation of the role of adjustment weights for estimate accuracy, I compared survey estimates with only design weights applied (when available) against survey estimates with both design weights (when available) and adjustment weights applied. Figure 6 shows the estimates for absolute accuracy compared with the mean benchmark value over the fieldwork period. The differences in estimates with and without adjustment weighting indicate that for 18 of the 23 surveys, adjustment weights improved accuracy. Measured by the mean absolute accuracy, the estimates with adjustment weights were 2% points more accurate than those without (absolute means = 9 and 11% points, respectively). The robustness checks reached the same conclusion regarding the positive impact of adjustment weights on accuracy (Fig. S9 and Fig. S10 in the supplementary material).Fig. 6. Absolute accuracy of survey estimates of the COVID-19 vaccination rate with and without adjustment weighting compared with the mean benchmark value over the fieldwork period, age group 18–59 years

Figure 7 presents the difference in estimates with and without adjustment weighting for the age group 60 years and older. Similarly to the age group 18–59 years, estimates with adjustment weights were more accurate in 30 of the 39 surveys included in the analysis. The mean difference in absolute accuracy of 1% point in the age group 60 years and older was smaller than that in the younger age group (absolute means = 6 and 7% points, respectively). Robustness checks confirmed the positive impact of adjustment weights on survey accuracy (Fig. S11 and Fig. S12 in the supplementary material).Fig. 7. Absolute accuracy of survey estimates of the COVID-19 vaccination rate with and without adjustment weighting compared with the mean benchmark value over the fieldwork period, age group 60 years and older

Discussion

Surveys are an important source of population health data. Compared with administrative data, they have the advantage of providing comprehensive information on the individual level. However, accuracy is a prerequisite for the reliability of results derived from surveys. To advance research on the data quality of health-related survey measures, the study evaluated the accuracy of survey estimates of the COVID-19 vaccination rate in Germany. For this purpose, I compared the survey estimates with administrative benchmark data (RQ1) and examined whether estimate accuracy differed by survey design (RQ2) and was improved by adjustment weighting (RQ3).

For RQ1, the results showed that the accuracy of survey estimates varied systematically over time. Whereas most surveys conducted in the first months of the vaccination campaign underestimated the vaccination rate, later surveys mostly overestimated it. This pattern indicates a shift in the dominant sources of error over the course of the campaign. The transition from under- to overestimation corresponds directly with the vaccine unrestricted availability for the general population in each age group, with cut-off months of June 2021 for those aged 18–59 years and April 2021 for those aged 60 years and older (see Fig. 1).

The underestimation of the vaccination rate in the early surveys may be explained by the prevalence of representation error, specifically nonresponse bias. During this initial phase, vaccine eligibility was restricted to priority groups, such as the elderly, persons with underlying health conditions, and those in long-term care facilities. These groups were likely more difficult to reach with general population surveys. Consequently, early surveys disproportionately represented the large segment of the population that was not yet eligible and therefore unvaccinated, leading to an overall underestimation of the vaccination rate when compared to the administrative data.

The overestimation that I observed in most surveys in my sample (and especially in those conducted in the later months of the vaccination campaign) aligns with the findings of similar studies on the vaccine uptake in other countries or regions, for example, the United States [46], Sub-Saharan Africa [47], and India [48]. This systematic overestimation can be attributed to multiple types of error.

First, measurement error, particularly social desirability bias, likely played a significant role. As vaccination became a widespread social norm, respondents who were not vaccinated may have untruthfully reported that they were, in order to align with a perceived social expectation. This hypothesis is supported by the results of a survey experiment conducted in Germany during the vaccination campaign [22].

Second, representation errors persisted and contributed to the overestimation. Nonresponse bias due to self-selection was likely present as individuals with a positive attitude toward vaccination may have been more willing to participate in the surveys than those who were vaccine hesitant. Unfortunately, the full extent of this bias typically cannot be measured based on the survey data. Additionally, the language barrier may have been another source of nonresponse bias, as persons without knowledge of German were excluded from surveys that did not offer additional language options. If survey participation correlated with the vaccine uptake in that group, bias may have arisen. This issue was pointed out by the Robert Koch Institute, which analyzed the difference between the vaccination rate estimated based on their administrative data and on a survey they conducted [49].

For RQ2, the analysis revealed differences in accuracy by survey design group – a variable defined based on a combination of sampling procedure and survey mode. Among the survey design groups in my sample, the probability mixed-mode (PAPI & CAWI; CATI & CAWI) surveys and the probability CAPI surveys achieved the highest average accuracy. Although some of the nonprobability surveys also provided comparatively accurate estimates, others did not perform well, resulting in an overall high variance in this group. Rohr et al. [15] found a similar pattern among nonprobability surveys. Overall, my findings based on a large-scale evaluation of individual survey design groups complement the literature on the data quality of health-related surveys, which has so far focused mostly on comparisons of a limited number of survey modes [50–52] or between probability and nonprobability sampling procedures [53–55]. However, further research on the differences between survey design groups is needed, as my analyses do not allow for the identification of determinants of accuracy at the level of the component variables – that is, survey mode and sampling procedure.

Investigating RQ3, I found a positive impact of adjustment weighting on the accuracy of the survey estimates of the COVID-19 vaccination rate. In other words, the sociodemographic variables used to create the adjustment weights in the respective surveys correlated sufficiently with both survey participation and vaccination status. This result is in line with findings from other studies that have demonstrated that weighting enhances the accuracy of various sociodemographic and health-related estimates [25–28]. Given the wide use of adjustment weights in analyses of health survey data [56], I consider further research on the effectiveness of adjustment weighting for specific outcome variables to be necessary.

The study is not without limitations. First, I relied on administrative health data as the best available benchmark, which itself may be prone to errors. In particular, the quality of administrative data on COVID-19 vaccination rates depends on accurate and complete reporting by the institutions responsible for vaccine administration. During the vaccination campaign, the Robert Koch Institute, the publisher of the administrative data on the COVID-19 vaccination rate in Germany, stated that the vaccination rate based on administrative data might be an underestimation of the real rate [49, 57]. Possible reasons for the underestimation included incomplete reporting of one vaccination type by contract physicians (leading to undercounts in certain age groups), limited reporting by occupational physicians, and incomplete transmission of vaccination data from medical practices to the central registry. Although my analyses were based on the most recent and best available version of the administrative data, including corrections made by the Robert Koch Institute, I emphasize the need for caution when interpreting my findings.

Second, the two data types — administrative and survey data — may not fully cover identical populations, which could partly account for differences in vaccination rate estimates. To mitigate this risk, I restricted the analyses to surveys targeting the general resident population in Germany, where vaccination was available to all individuals living, working, or insured in the country regardless of citizenship, and excluded surveys with narrower target groups. Nonetheless, some minor coverage dissimilarities may remain.

Third, the utilized dataset with coded information on surveys does not contain metadata on either the position of the COVID-19 vaccination question within each questionnaire or the overarching survey topic. However, prior research demonstrates that both the placement of survey questions and topic salience can influence data quality through errors related to representation [58–60]. Future research could therefore extend the analysis by examining the role of survey topic (general or COVID-19 specific) and question placement on the accuracy of the survey estimates.

Conclusions

In this study, I evaluated the accuracy of survey estimates of the COVID-19 vaccination rate in Germany through a comparison with administrative benchmark health data. Based on the results, I conclude that population estimates of the vaccination rate derived from surveys should be interpreted with caution, as inaccuracy — and particularly overestimation — was widely prevalent in the surveys examined. Furthermore, I emphasize the importance of survey design for accuracy. Specifically, I have shown that probability mixed-mode (PAPI & CAWI; CATI & CAWI) and CAPI surveys provided more accurate estimates, on average, than did other designs. Finally, I encourage the use of adjustment weights, as the findings indicate that they have the potential to improve accuracy, as demonstrated by the substantial improvement in most vaccination rate estimates in my study.

Supplementary Information

Supplementary material 1.