Using smartphones to study vaccination decisions in the wild

TL;DR

This study uses smartphone-based games to explore how people make vaccination decisions in a realistic setting.

Contribution

A novel smartphone-based gamified method is introduced to study vaccination decisions with real-world disease spread dynamics.

Findings

Participants prioritized information about their own health outcomes over others' decisions.

The gamified smartphone approach successfully simulated disease spread and vaccination dynamics.

Multiple rounds of the game provided insights into individual decision-making processes.

Abstract

One of the most important tools available to limit the spread and impact of infectious diseases is vaccination. It is therefore important to understand what factors determine people’s vaccination decisions. To this end, previous behavioural research made use of, (i) controlled but often abstract or hypothetical studies (e.g., vignettes) or, (ii) realistic but typically less flexible studies that make it difficult to understand individual decision processes (e.g., clinical trials). Combining the best of these approaches, we propose integrating real-world Bluetooth contacts via smartphones in several rounds of a game scenario, as a novel methodology to study vaccination decisions and disease spread. In our 12-week proof-of-concept study conducted with N = 494 students, we found that participants strongly responded to some of the information provided to them during or after each decision round, particularly those related to their individual health outcomes. In contrast, information related to others’ decisions and outcomes (e.g., the number of vaccinated or infected individuals) appeared to be less important. We discuss the potential of this novel method and point to fruitful areas for future research.

Vaccination is a safe and effective method to eliminate infectious diseases. Therefore, understanding what determines vaccination decisions is important to better communicate about vaccination and develop interventions that support vaccine uptake. Current methods to examine individual vaccination behavior, however, suffer from different problems. Highly controlled lab or survey studies, one the one hand, often ignore real-world diseases’ spreading dynamics. On the other hand, clinical trials in the field are often not able to provide insights into this decision process. We present a novel method that provides high control and insights into the decision processes, while also considering the dynamics of real-world disease spread and vaccination. Specifically, using a smartphone application and a gamified approach, we simulate the dynamics of disease spreading via Bluetooth contacts. We construct a point system to simulate, for instance, disease symptoms and vaccine side-effects. In our proof-of-concept study, we conduct multiple rounds of this interactive vaccination game, and give players information about infected, recovered, susceptible, and vaccinated individuals in their environment, to investigate how this affects their own vaccination decisions. We find that players give the most value to the information regarding their own experience in the game, rather than the environmental information of others’ behaviors.

Figures

Vaccination Probability given Point Loss in Former Round and Previous Decision.Effects of point loss in the former round and the previous decision on the probability to get vaccinated in the subsequent round (k = 3,934 observations by n = 469 players). Dots represent the estimated probability and error bars the standard error. We see that a point loss implies a lower vaccination probability, but only for players that did not vaccinate in the previous round.

Survival model outcome.Panel A shows the survival curves (i.e., probability of a player to be unvaccinated at a specific day) in the two feedback conditions (average over four rounds) from day 2. The cumulative number of unvaccinated players from day 2 are shown in Panel B. Panel C shows the Hazard Ratio (i.e., likelihood of getting vaccinated) of daily feedback about the share of susceptible others in the local vs. global feedback condition (k = 2,003 observations by n = 382 players). We see that, for the Global Feedback condition, the likelihood of getting vaccinated exponentially increases when the number of susceptible individuals is high.

Finite state automaton of the experiment.All players start from the susceptible state and they can remain so, decide to get vaccinated or get infected and then recover. Labels on the transitions show what event is triggering them in the partly probabilistic-causal game framework.

Full text

Introduction

Vaccination is an effective and safe prevention measure to protect people from infectious diseases, saving 2–3 million lives worldwide every year [1]. In the special case of the COVID-19 pandemic, it has been estimated that vaccines prevented about 20 million deaths during the first year of COVID-19 vaccination only [2]. Despite this success, vaccine hesitancy—i.e., delay in acceptance or refusal of vaccination despite the availability of vaccination services—poses a major threat to global health and addressing it is therefore a priority area for national and international health organisations [3–5]. Accordingly, building an understanding of why people decide in favor or against vaccination is a crucial challenge for our times.

For recommended vaccinations, the personal benefits (e.g., reduced risk of infection) are expected to exceed the potential costs (e.g., risk of vaccine-adverse events). In addition, most vaccines also provide social benefits due to herd immunity [6]. That is, as a population’s vaccination rate increases, the spreading of the pathogen is reduced, such that non-vaccinated individuals are indirectly protected. Therefore, someone’s vaccination may also benefit others who either cannot get vaccinated (e.g., infants, immunocompromised persons) or do not want to get vaccinated (e.g., vaccine-hesitant persons). Considering both its personal and social benefits, vaccination poses a strategic interaction, in which individuals have to decide whether to prosocially benefit non-vaccinated others (accountable or unaccountable) by getting vaccinated or whether to free ride on the indirect protection provided by vaccinated others, resulting in a social dilemma [7–9].

Given the potential role of personal and social benefits, the vaccination decision should depend on both (i) internal states, such as personal attitudes, perceptions and expectations, and experiences related to the disease and to the vaccine, as well as (ii) external data, such as information about others’ vaccination behaviour, vaccine uptake in a population, or the incidence of diseases [5, 10–12]. Regarding the latter, previous research has indeed shown that people consider both the decisions of others, and the welfare consequences for others when making their own vaccination decisions. However, the evidence regarding the direction of such effects is mixed. Some studies indicate that informing people about the social benefits of vaccination increases their vaccination intentions [8, 13–15]. In contrast, other studies provided evidence that free riding may occur when people learn that they are protected via herd immunity [16–18]. Moreover, further studies showed that people increase their vaccination intentions when they believe that others are (planning to get) vaccinated, too, indicating adherence to perceived social norms [19–21].

Spatial and temporal dynamics in vaccination decisions and their consequences

The mixed results regarding the impact of social information and its consequences on individual vaccination decisions may be partly attributed to the heterogeneity in methods used in prior research. That is, depending on the specific research question, different studies have increased the salience of certain contextual cues, emphasizing some information (e.g., social norms [10, 20], social benefits and its beneficiaries [9, 13], varying personal benefits depending on the amount of vaccinated others [8, 15]) while omitting others. Such differences in the salience of certain cues emerge easily, because when laboratory and online studies use artificial or imagined vaccination decisions, they ignore the spatial and temporal dynamics of vaccination decisions and their potential consequences as they occur in reality. That is, first, vaccination decisions in studies are typically independent from these dynamics, but real diseases are transmitted via some form of (direct or indirect) physical contact between individuals (i.e., spatial dynamics). Specifically, transmissions through direct contact can only take place when, at the time of contact, one individual is infectious and the other one is susceptible. Second, a vaccination decision temporally precedes its consequences, while personal and social information, such as experiences from own or others’ previous vaccinations, may be available and updated over time (i.e., temporal dynamics).

Neglecting these spatial and temporal dynamics of vaccination decisions and their consequences is a major limitation of prior research’s external validity. An obvious solution is to study actual vaccination decisions and their consequences ‘in the wild.’ However, a severe drawback of this approach is that the decision processes, such as what kind of personal and social information is available to people in their decision making, are often unavailable to researchers. Here, we aim to overcome these limitations and increase external validity by presenting a novel experimental design that combines the best of both approaches, that is, studying the determinants and consequences of vaccination decisions while also considering the spatial and temporal dynamics as they unfold in the real world.

Using smartphones to study vaccination decisions and infection spread

We use an established interactive vaccination game that builds on a SIR (Susceptible-Infectious-Recovered) model of disease dynamics, thus capturing the dynamic personal and social benefits of vaccinations depending on the population’s vaccination rate [16]. Various versions of this interactive vaccination game (I-Vax) have been successfully applied to investigate vaccination behaviour and potential behavioural interventions, with games played several rounds having usually taken place in controlled lab-settings and one-shot versions as online studies [22–28]. The key advantage of this gamified method is that decisions are not hypothetical, but they have actual consequences by providing behaviour-contingent monetary incentives, thus reducing the intention-behaviour gap [29].

Here, we go a step further by implementing the I-Vax game in a natural scenario, taking into account the spatial and temporal dynamics of vaccination decisions and their consequences. We give players augmented information on the natural environment through the game mechanics, which are based on the real-world experiences of the participants. In our approach, we study a cohort of nearly 500 individuals, all freshmen at a Danish university, where infections take place via Bluetooth contact between participants’ smartphones—that is, infections can only happen when two participating individuals are physically close one another (infection range up to ten meters [30]). This models the spatial properties of real-world disease spreading, i.e., infections happen because of actual physical contact between infectious and susceptible individuals. Through the implemented application the participants could take the decision to vaccinate and protect themselves from the fictional disease. Our I-Vax study, took place across an entire 13-week semester of students attending a single campus, thus generating millions of contacts [31].

We implemented 12 independent game rounds, each lasting one week, where players had their naturally occurring and repeated physical contacts with each other, to model the temporal properties of vaccination decisions and their consequences. Across the 12 rounds, we varied the information available to players ranging from no information in the beginning of the game, via local information about the vaccination status of neighbors, to full population-level information in the final rounds of the game (see the subsection below for full details). As each information setting was deployed multiple times, participants could learn from previous rounds and potentially change their perceptions of the disease and vaccination as well as their beliefs about other participants’ behaviours over time. This mirrors repeated vaccination decisions, such as in the case of influenza or COVID-19 (i.e., booster doses). As such, our set-up allows a ‘time travel’ through a repeated vaccination history under behaviour-contingent incentives that would otherwise require years of observations ‘in the wild.’ Importantly, given that all I-Vax decisions to vaccinate or not were made via a smartphone application, we have access to all the information that participants received during the game, and can therefore examine the impact of all pieces of information on players’ vaccination decisions.

In summary, in this study we combine the benefits of experimental control with a more valid setting: the study’s set-up allows controlling for or tracking numerous factors related to decision makers’ personal and social benefits of vaccination as in artificial, but highly controlled vaccination studies. At the same time the set-up can be considered to increase external validity due to the mimicking of spatial and temporal dynamics of disease spreading in natural conditions, even though the nature and magnitude of costs and benefits in the gamified setting still differ from actual physcial consequences due to illness or side effects.

Study set-up and data handling

Our exploratory proof-of-concept experimental study was conducted with N = 494 students at a Danish university as part of the SensibleDTU project [31]. Participants were given a smartphone which recorded sensor data and usage data. This allowed the implementation of the I-Vax game, including the use of Bluetooth sensors for proximity detection and disease transmission. In this section, we describe an overview of the basic set-up and the detailed factors that were exogenously varied during the experimental study. We provide more details on the parametrisation and implementation in the Methods section.

Game mechanics

The game mechanics were designed to recreate the vaccination process and the spreading of the disease to be as similar as possible to natural conditions. Some participants were infected at the start of each round and thus treated as ‘patients zero’, or ‘seeds’. These players then spread the disease to physically close susceptible players, who were neither vaccinated, infected nor recovered. The infection underwent a phase of simulated incubation of the disease. Thus, players did not become infectious immediately following the infection, but only after a fixed amount of time. They also recovered from the disease after a fixed amount of time. Players could decide to get vaccinated at any point in time during the round. As under natural conditions, the vaccination was not immediately effective, but it became so after eight hours. In our implementation, the vaccine is 100% effective.

In each round, players started with 100 points. Points are deducted to simulate fixed vaccination costs (e.g., the effort to make an appointment with the doctor and the time spent receiving the jab), vaccination side effects, and the effects of getting sick with the disease when unvaccinated. Importantly, whereas vaccination costs are low, fixed and certain, the probability and severity of vaccination side-effects follow an exogenous probability function, and disease costs are determined by the endogenous probability of having a (Bluetooth) contact with an infectious player coupled with an exogenous severity function (for a detailed rationalisation, see [16]). Moreover, the average point loss due to encountering the disease outweighs the side-effects vaccination loss. Finally, to motivate players to make rational decisions, we used the point system to track individual-level performance, which determined rewards at the end of the experiment—the players with the most points after the full 12 rounds semester won iPads and FitBits.

Manipulations

We vary the information environment in a within-participants design with a fixed order. All information is available to participants through their smartphone app. In rounds 1–4, players only rely on their personal attitude and experience in the game (no feedback condition). In rounds 5–8, players receive information about their surrounding environment (local feedback condition). This means that every day they learn about the amount of infected, vaccinated, and susceptible players among the players they interacted with on the previous day. Finally, in rounds 9–12, instead of receiving the statistics of their contacts, players receive information about the whole population (global feedback condition). At the end of each round, players also get the cumulative information for the whole round, in both the local and global feedback conditions. We define these as the different conditions of the environmental feedback we give them.

With these different information environments, we can explore whether the social information available in the spatial environment affects participants’ vaccination decisions. We modeled the individual vaccination decisions using the information cues available to the players (e.g., share of infected, vaccinated, and susceptible players) as factors. Therefore, we are able to distinguish between different information cues (e.g., share of infected, vaccinated, and susceptible players) and information effects as suggested by previous laboratory and online research (e.g., prosocial vaccination, free riding, social-norm following). The realistic set-up, which considers the spatial and temporal dynamics of natural vaccination decisions and infections, allows for the estimation of these effects without sacrificing data availability and experimental control.

Analytical approach

As a first step, we analyzed players’ vaccine uptake across rounds, only considering the individual experience they gained by playing consecutive rounds. This implies drawing on information of their performance at the end of each round, i.e., whether there was a point loss and whether it was due to potential side effects in case of vaccination or due to potential infection with the disease in case of non-vaccination. As a second step, we added the environmental information that were given to players in rounds 5–12. The analysis rests on different feedback about other players’ behaviours and outcomes at the end of each round. The final step consisted of adding the time component (day) interior to each weekly round, to see how the effects of different factors change when considering a more dynamic setting, including information received within each round.

For the first two steps, Mixed Linear Effects (MLE) models [32] have been used for the analyses, considering each round as a ‘box’ and, thus, reflecting the analysis that is more commonly done in laboratory and online experiments and even in many real-world clinical trials. For the final step, Survival Models [33] were used, as they were the optimal method to promote the benefit of our experiments’ temporal and spatial resolution within each round.

This study is meant as a proof of concept for the deployment of virtual epidemics and their application for studying vaccination behaviors. This pilot study suffers from some drawbacks, most notably some rounds of disease dying off due to pure chance and the declining engagement over the time course of the study (see the Discussion section for more details on these issues). Therefore, we do not make any claims about descriptive (mean level) differences in vaccine uptake, but we rather focus on individual factors of vaccine uptake. All the analyses should be treated as exploratory. We provide a full description of the specific methods underlying the presented results in the Methods section.

Results

Players’ behaviour given individual outcome in previous round

As described in the Introduction, the first analysis involves predicting vaccination decisions across rounds by using individual outcomes in the previous round. The results are shown in Table 1. The model was constructed so that the intercept represents a baseline setting, where players only received individual feedback, they did not have a point loss (neither from the disease nor a vaccination side effect), and they did not get vaccinated in the previous round. Finding a negative coefficient shows that, overall, the decision to avoid vaccination was more common than getting vaccinated. The other factors are dummy variables that represent the three different feedback conditions (no feedback, local feedback, global feedback) described above and whether or not there was a point loss. We included an interaction term between point loss and the previous decision, which allowed us to control whether the point loss came from the disease or from vaccination side effects.

Table 1: MLE model predicting vaccinations decisions across all rounds by feedback of personal consequences after each round (k = 3,934 observations by n = 469 players).

We found positive effects of Point Loss and Previous Vaccination Decision, qualified by a significant interaction effect. As shown in Fig 1, players who did get vaccinated in the previous round were equally likely to get vaccinated in the next round, irrespective of whether they experienced vaccination side effects or not. In contrast, players who did not get vaccinated in the previous round were more likely to get vaccinated subsequently when they got sick (and lost points) than when they did not experience disease costs and thus were successful free-riders.

Vaccination Probability given Point Loss in Former Round and Previous Decision.Effects of point loss in the former round and the previous decision on the probability to get vaccinated in the subsequent round (k = 3,934 observations by n = 469 players). Dots represent the estimated probability and error bars the standard error. We see that a point loss implies a lower vaccination probability, but only for players that did not vaccinate in the previous round.

This result indicates that players responded in a rational way to negative outcomes, probably to maximize their outcomes. Interestingly, players appear to be more sensitive to the costs of infection than to the costs of vaccination. The analysis shows further (negative) effects of Round Number and Global Feedback, indicating that players were less likely to get vaccinated in later rounds, and that providing global feedback about others’ decisions and outcomes also reduced their own likelihood to get vaccinated. Taken together, our findings suggest that players were sensitive to the outcomes of the game, providing a first proof-of-concept for the study setup’s validity.

Players’ behaviour given environmental feedback

To analyze the impact of the environmental feedback on players’ vaccination decisions, we distinguish between the following pieces of information shown to each player at the end of each round in the local and global feedback conditions: share of vaccinated players (Vaccinated EndOfRound), share of infected players (Infected EndOfRound) and share of susceptible players (Susceptible EndOfRound). The baseline setting is now the local feedback condition and the main effects of the feedback factors refer to their influence in this condition only. Therefore, we also added interaction terms between the information players received and the feedback condition, capturing the impact of information in the global feedback condition.

As shown in Table 2, the feedback on other players’ behaviours and outcomes did not significantly affect their own vaccination decision, in any of the feedback conditions. In contrast, the information on personal outcomes remains to significantly affect players’ decisions, as reported above. This is surprising, considering how players behaved differently following the feedback settings (as demonstrated in the previous subsection). Our conclusion in this regard is that participants, in our experiment, value much more the individual feedback rather than the environmental feedback—another interpretation could be a lack of interest in the game over time, which we cannot rule out due to the fixed order of the feedback conditions.

Table 2: MLE model predicting vaccinations decisions across rounds 5–12 by feedback of personal consequences and environmental feedback after each round (k = 2,481 observations by n = 427 players).

Players’ behaviour given daily feedback

The final part of the analysis regards the most unique part of our methodology, which adds the time dimension within each round. As in a natural setting, players received daily feedback on whether other players had been vaccinated, infected, or susceptible. Specifically, at the start of each day, they receive information, relative to the previous day, about their environment and either on other players they have been in contact with (local feedback condition) or on all other players in the population (global feedback condition). We use this information shown to participants to predict their likelihood of getting vaccinated within a Survival Analysis model [33], with the Hazard Ratio indicating the (increased or decreased) likelihood of getting vaccinated over time. Given the structure of the daily feedback, the analysis is using the players’ behavior starting from day two and lasting for the remaining 6 days of the round.

Our first observation is that 84.4% of the vaccinations over all rounds happened during the first day of the week. This may indicate that players were strongly affected by the feedback they received at the end of the rounds (as analysed above), rather than the feedback they received during the progress of the rounds. The survival model is estimated on the remaining share of vaccination decisions that were not made on the very first day of a round. The survival curve showing unvaccinated players is displayed in Fig 2, Panel A, indicating how the probability of not vaccinating decreases after the initial vaccinations on day one (not in the plot). The probability of not getting vaccinated decreases with a very slow rate and remains quite high at the end of the week. This is expressed in numbers (aggregated over all rounds of the same setting) in Fig 2, Panel B.

Survival model outcome.Panel A shows the survival curves (i.e., probability of a player to be unvaccinated at a specific day) in the two feedback conditions (average over four rounds) from day 2. The cumulative number of unvaccinated players from day 2 are shown in Panel B. Panel C shows the Hazard Ratio (i.e., likelihood of getting vaccinated) of daily feedback about the share of susceptible others in the local vs. global feedback condition (k = 2,003 observations by n = 382 players). We see that, for the Global Feedback condition, the likelihood of getting vaccinated exponentially increases when the number of susceptible individuals is high.

The specific model is Cox Proportional Hazard (CoxPH) [33], which, similarly to MLE, explains the impact of each factor on vaccination decisions. The results are shown in Table 3. As in the previous analysis, the baseline setting is the local feedback condition, that is, feedback about other players’ decisions and outcomes with whom one had physical interactions as assessed through Bluetooth contacts. There is a positive effect of the Previous Vaccination Decision. This indicates that even those players who did not get vaccinated on the very first day of a new round were more likely to get vaccinated later during the round when they were already vaccinated in the previous one. The main effects of the Global Feedback condition and the daily feedback on the number of susceptible players, i.e. Daily Susceptible, were qualified by a significant interaction effect between these variables. As shown in Fig 2 Panel C, players were more likely to get vaccinated when they received daily feedback about an increasing share of susceptible others, but with a bigger difference in the global setting than in the local setting. This appears reasonable, given that if many people are at risk, the urge to protect oneself could increase. In absolute numbers, in the global setting, the likelihood of getting vaccinated really has a spike when about 75% or more of the players in the general population were susceptible, reaching an Hazard Ratio of ≈ 0.15.

Table 3: Survival Model predicting whether a vaccination happens before the end of the round across rounds 5–12 by feedback of personal consequences and environmental feedback after each round (k = 2,003 observations by n = 382 players).

Discussion

The main goal of the present research was to devise and gain seminal experience with a novel method to study vaccination decisions and their consequences in an experimental set-up that matches real-world infections more closely than in previous research, while retaining a high level of experimental control. Specifically, whereas previous experimental research on vaccination decisions typically ignored the spatial and temporal dynamics of vaccination and disease spread, we implemented a scenario in which people may contract a fictional disease by Bluetooth contact with other players in the real world. The use of a smartphone application allowed the spreading of the disease to occur in physical space, following time patterns for the events (vaccination, incubation time, etc.) that reflect real-world mechanics. To make the decisions and outcomes meaningful, we implemented an interactive vaccination game with behaviour-contingent incentives [16].

To validate this novel approach, we conducted a comprehensive experimental investigation involving N = 494 participants over a span of 12 weeks. Throughout this period, which was divided into rounds in a gamified fashion, we manipulated the information given to participants regarding other players’ choices on vaccination and their consequences. Overall, our results suggest that players were sensitive to the feedback and associated outcomes. For instance, we found that players increased their likelihood of getting vaccinated when they got infected in the previous round. Moreover, we found some evidence of players reacting to other others’ behaviors, too. Particularly, players who did not decide immediately to get vaccinated, were later more likely to get vaccinated when they saw higher numbers of susceptible players in subsequent rounds. This could indicate that players perceived either a potential danger that the infection could easily reach them, or some prosocial motivation to protect others with one’s own vaccination [9]. Taken together, both selfish rationales, as well as prosocial motives, could account for the observed behaviours.

The feedback manipulation was a novel aspect of the experimental case study with our I-Vax game. We found some differences in players’ responses to feedback from interactions with players vs. feedback about the whole study population. For example, participants valued more their own experience and did not consider much the feedback given at the end of the round, be that global or local. They, instead, valued highly the daily information about the number of players that were still susceptible. This observation deserves attention in future research. Although we implemented the variation of local vs. global feedback mainly for exploratory reasons, future studies could investigate the role of social norms, free-riding motivation, and prosocial concern when people learn about a more local and potentially psychologically closer group versus a more distant but larger group. For example, in geographically secluded areas with small communities, it is possible to reconstruct the infection network if information about individual vaccine uptake and disease cases are publicly known. Previous research showed how in such communities there might a greater willingness to contribute to herd immunity by getting vaccinated [15]. Our study set-up and feedback manipulation can be considered an approximation of such a case. Future research may therefore use this set-up to better understand the network dynamics underlying local vs. global feedback. Clearly, differences in how people value and react to such information could inform policymakers to improve vaccination communication.

Another interesting effect to investigate would be the effect of individual attitudes towards vaccinations in our experiment. We conjecture that feedback might have a bigger impact on players with a rather low or medium vaccination attitude, as those with a high vaccination attitude would vaccinate nonetheless. In fact, we see that most of the vaccinations happen during the first day of the round, indicating how important the individual attitudes might be.

Limitations

Although the method utilized here is powerful, the experimental set-up had some limitations that should be considered when interpreting our findings and planning future research endeavors. First, in some instances of the local feedback condition, a scarcity of initially infected seed users may have created a misleading perception of reduced disease transmission, potentially leading to lower vaccine uptake in subsequent rounds. Fortunately, this issue had a limited impact on our study, as vaccinations showed a consistent decline over time, rather than a sudden drop after these rounds. Moreover, the study of each of the conditions as the aggregation of their relative four rounds mitigated the drawback on our analysis. Nevertheless, we recommend cautious seed user selection in future implementations, for example, by selecting a fixed fraction of the population to start the round as infected instead of assigning a probability of them starting as initial seeds.

Second, we acknowledge that certain parameter conditions in our set-up, such as 100% vaccine efficacy, a 0.3 probability of vaccination side effects, and uniform risk categorization among users, do not mirror real-world scenarios. However, the parametrization of I-Vax game’s set-up allows for easy adjustment of these parameters.

Following up on the parameters and settings of our method, we chose seasonal influenza as a model of our fictional disease, which typically results in only mild symptoms when contracted. This means that, differently from a pandemic scenario, users are not encouraged or forced to self-isolate, allowing us to model asymptomatic spreading of the disease as well. While the absence of this self-isolation can be a limitation of our methodology, previous research also showed that self-removal can be not as common, and that, instead, the phenomenon of work presenteeism [34] (i.e., going to work even when symptomatic) can have a significant impact on the diffusion of the disease.

Another difference from a pandemic scenario is the magnitude of disease symptoms and vaccination side-effects, which may differ significantly from our scenario. In fact, the vaccination costs could be substantially higher, both in the administration, in the side-effects, but also because of the cost of the social isolation deriving from quarantining. On the same note, the severity of the disease may show a greater difference of consequences, from very mild to extreme symptoms, either making the costs of vaccinating non-important or outweigh the costs of the disease.

Given these observations, paying more attention to these aspects of seasonal diseases or pandemic scenarios, results in relevant research questions that could be tackled by adapting the parameters of our methodology, as we presented them, to the desired scenario. For example, conditions with and without social costs of self-isolation could help in understanding its impact from a mental well-being point of view.

Next, in our set-up we consecutively implement three different conditions, in which players have different environmental information to consider for their vaccination decision. This influence was evident in the decreasing vaccination rates over time. For future implementations, introducing user groups subjected to different conditions simultaneously could enhance result robustness.

Involving many participants for a study such as this can be demanding in terms of recruiting users. This can still be facilitated by the widespread presence of smartphones, onto which users could just install the application. While studying a population well apt with digital tools, such as university students, can help in this issue, it still has the drawback of not being representative of the whole population. Furthermore, concerns might arise regarding users’ data protection, which we suggest to handle with attention (also discussed in the Methods section).

Finally, proper to the gamification approach, dropout can be a concern. Fortunately, our study experienced relatively low dropout, potentially due to economic incentives driving player engagement. Nevertheless, the inconsistent participant numbers across rounds introduced some variability, which we have mitigated through the adaptation of the applied models.

Conclusion

In summary, our novel method combines the best of highly controlled laboratory and online experiments, and large-scale clinical trials studying actual vaccine uptake. We propose that running experiments using this method could provide important insights before rolling out said clinical trials (e.g. to evaluate interventions aiming at increasing vaccine uptake), and potentially reduce the attitude-behaviour gap that plagues implementation research.

Methods

The first proof-of-concept implementation of our novel method was integrated into the SensibleDTU experiment [31]. SensibleDTU is a living lab experiment conducted at the Technical University of Denmark (DTU), in which 1000 students were provided a smartphone, and sensor data was collected, along with surveys and app usage information.

In this setting, the I-Vax game was implemented as a smartphone application. This implementation had several advantages. Firstly, the modelling of the disease spreading followed real-life mechanics. We set the R0 parameter to 3, to mimic the spread of seasonal influenza [35] (meaning that an infected person on average spreads the disease to 3 others). The spreading follows physical contact between the participants, using Bluetooth technology [30].

Gamification is another fundamental aspect of our novel methodology, as the participants are incentivised to behave responsively to the game with economic incentives, based on an in-game point system. This, in conjunction with the time and space components of the game, reconstructed a natural scenario, both for the participants’ experience and for disease transmission.

I-Vax game implementation

The modelling of the game, as described above, is composed of both game design and disease modelling. In the following paragraphs, we will describe in detail our implementation. We report summary statistics in S1 Appendix.

Point system

A point system was devised to guide players’ decisions and make them understand the outcome of the game and evaluate their actual performance. Players started each round with 100 fitness points. Following specific events, they could lose points, simulating real-life consequences and costs:

Vaccination: Real vaccinations often require spending time and effort (e.g., to book the appointment and to actually get the injection), and they may also cost actual money. To simulate this, players had to pay 10 points to get vaccinated. The vaccine was 100% effective.Vaccination side effects: It is possible to have allergic reactions or other side effects after a vaccination. To simulate these events, a vaccinated player had a 0.3 probability of getting side effects and losing points: this probability allows to control for rare severe consequences and common very mild symptoms like arm soreness [36]. To account for this, the point loss could range from 0 to 90 points (on top of the vaccination cost), with an average of 30 points lost.Infection: Different symptoms are possible in real-world diseases, but the consequences of getting infected are usually worse than possible vaccination side effects. Thus, when infected, the point loss still ranges in the 0–90 points interval, but with a higher average of 50 points lost. Low values of point loss are included to simulate asymptomatic infections or those with very mild symptoms.

Finite state automaton

To model the states of the players during the game and the transitions between the states, we constructed a finite state automaton, represented in Fig 3. The essential states are: Susceptible (S), Vaccinated (V), Infected (I) and Recovered (R). Other states (Awaiting vaccination (A), Awaiting vaccination Exposed (AE) and Exposed (E)) were introduced to model specific mechanics and to manage the state transitions. At the start of a round, every player is susceptible. They can either remain so until the end of the round (if they do not get vaccinated or infected), they can get vaccinated to become immune to the disease, or they can get infected when being close to someone who is already infected (i.e., they are infectious). Players who got infected, heal from the disease after a set amount of time, becoming recovered (and immune).

Finite state automaton of the experiment.All players start from the susceptible state and they can remain so, decide to get vaccinated or get infected and then recover. Labels on the transitions show what event is triggering them in the partly probabilistic-causal game framework.

Vaccination mechanics

The core mechanics are vaccination and infection, which are sets of transitions. The vaccination is the only transition that players can trigger with their own decision. The vaccine is not effective immediately, as it is in real life, the body has to respond to it and register its information. In the same way, in our implementation, the vaccine is effective (at 100% effectiveness) after ca. 8 hours. The player has to pay the 10 points immediately (as he would spend time and money to have the vaccine shot, before achieving immunity). To model the time interval when the vaccine is not yet effective, a state was added, i.e., awaiting vaccination (A). During this time, the player was not protected from the virus and could get infected, transitioning to the state: awaiting vaccination exposed (AE), which then leads to an infection. Only after the immunity was achieved the player could experience side-effects, finally reaching state V.

Infection mechanics

To start the spread of the disease, with a probability of 0.015 players were chosen as initial seeds and their state was set to infected (I). This simulates the fact that at the start of an epidemic, there are no externally imposed containment efforts. This strategy also gives control over the initialization of the experiment, but led to some issues as described in the Limitations section. The seeds, and the newly infected, could spread the disease to players in state S or A, so that the R0 index (i.e. how many players get infected on average by a single infected) was roughly equal to 3, simulating a seasonal influenza [35]. Therefore, the probability of contracting the disease upon contact with an infected player was set to 0.0225. We also simulate the incubation time, in which the person exposed to the disease is not yet infected: for ca. 6 hours, the player was in state exposed (E). After the incubation time, the player was in infected state I, now being infectious themselves, for ca. 18 hours. After this period of time, players would recover, being unable to get infected again (or to vaccinate), getting to state R.

Feedback settings

The final mechanics of the game are repeated measures, i.e., multiple rounds were played, simulating the seasonality of the disease, like in a real-world scenario. Twelve rounds (or waves) were played. We changed the feedback given to players (through the I-Vax application) every 4 rounds, allowing the study of different feedback conditions. In this way, we achieve the controllability possible in artificial experiments in our realistic setting. In our experimental setting, we controlled for information about the players’ surrounding environment, and implemented 3 treatment conditions:

Rounds 1–4: At the end of each of these rounds, we gave players individual-level information, consisting of the outcome of the round, i.e., remaining points, final state and final decision. We studied the impact of this information on the next round’s vaccination decision. This condition is called No Feedback, because of the absence of environmental feedback. This type of information was provided throughout all the conditions.Rounds 5–8: The feedback given to players involves their local environment, i.e. the players they had contact with. This is called Local Feedback condition. Every day (starting from day 2 of the week) players were given information about the players they had interacted with the day before, in the form of the percentage of infected, vaccinated or susceptible among them. The same information was given at the end of the round, aggregating the daily feedback.Rounds 9–12: Similar to the previous condition, players receive daily and weekly feedback about the percentage of infected, vaccinated or susceptible, but relative to the whole population of players instead of their contacts. This is called Global Feedback condition.

Communication with participants and privacy considerations

A core part of the experiment is the communication with the participants. There are many possible types of bias that can alter the outcome of an experiment depending on how this communication happens. One specific worrisome bias, when it comes to communication, is response bias [37, 38]. This describes the fact that participants may take certain actions just because they are part of the experiment or they are influenced by the communication with the researchers. To counter this kind of bias, in our experiment, the communication with players has been done solely through the use of the I-Vax application implemented for the game. In the application, players could find the rules of the game and all the feedback information they were given, either daily or at the end of the rounds. Moreover, they were also made aware of the effect of herd immunity in the game, i.e., with 70% of the players vaccinated the probability of the disease spreading would have been zero. In this way, it was possible to simulate how people receive and perceive information in a real-world scenario and to understand how they are affected and which information affects them the most.

The privacy of the participants of the game was strictly respected. They were assigned an identifier based on their SensibleDTU ID. Thus, for further information relative to these concerns we refer the reader to [39, 40]. In the data we share, these IDs have been further anonymised.

Analysis methods

Following the structure of our experimental setting, we identified as fitting analysis two models, i.e., Mixed Linear Effects (MLE) [32] and Survival Analysis [33]. The analysis part is important to validate in practice the validity of our method. The data has been normalized and centered before the analysis.

MLE models are one of the standards for analyzing data involving human behaviour. They consist of linear regression, controlling for different factors (fixed effects) while also allowing the grouping of observations (i.e., random effects) where there could be a dependency or common characteristics or, as in our case, repeated measures. Each player had to make repeated decisions in different rounds, thus being the random effects. In this way, we established the independence between observations pertaining to different players, being able to assess individual performances and outcomes. The fixed effects, instead, include all the information that was given to players (remaining points, status, previous decisions, feedback), so to see their effect on the participants’ decisions. We use the logit link function, as the outcome is a binary decision, which maps the output of the model into the interval [0, 1], resulting in the vaccination probability. These models have been used to map the rounds of the experiment, as it is not really possible to capture the flow of time inside them. Following this, we used as input the snapshots of the end of the rounds, by feeding the models the individual outcome and the weekly feedback given to players.

Eqs (1) and (2) describe more in detail the models described in the first two subsections of the Results. Here the models predict the decision of the players, grouping the observations by player with the random effect (1|user), with the main effects and the interactions effects (indicated by the column symbol).

$$[eqn]$$ $$[eqn]$$

For the analysis of the daily feedback, instead, a more time-dependent model was needed. Survival Analysis models time until a set event happens at time t (usually the event is death, hence the name survival). Using days as the time measure, allowed us to analyze what happens during a round and to use as factors the daily feedback that was given to players. Since the scope of this analysis was to look into which of the factors were significant towards the vaccination decision, rather than modelling how many days would pass before a participant decided to get vaccinated, the specific model chosen was Cox Proportional Hazards (CoxPH) [33]. This model identifies how much different factors contribute towards the event, thus, in our case, the vaccination. Finally, its interpretation is very similar to the MLE models, making the whole analysis comprehensive and coherent at the same time.

The following Eq (3) describes the survival model used in the third subsection of the Results. Similarly to above, it groups the users and uses both main and interaction effects of the factors we consider. The output is the final decision along with the day in which that decision would be taken (i.e., the time in which the event has happened). Note that we did not consider this in our results, as we consider most important the factors that foster the decision rather than its timing.

$$[eqn]$$

Ethical statement

The research project and data collection was registered with and approved by the Danish Data Supervision Authority before data collection commenced. All data was collected with informed consent and with every participant able to withdraw from the study and have their data deleted. This protocol, implemented in 2012 and 2013, was in effect similar to the rules being introduced with the EU General Data Protection Regulation (GDPR) which came into effect in May 2018. In the present release, to comply with GDPR, the data has been stripped of personally identifying information and the data has been reduced in such a way that there is no reasonable likelihood of re-identification occurring.

Supporting information

S1 AppendixRound statistics.Document containing statistics of infected, vaccinated and seeds of each round.(PDF)