An Overview of the Impact of an Embedded Evaluation of the Immunisation Education, Training and Clinical Support Programme During the New Zealand COVID-19 Pandemic

TL;DR

This paper discusses how an embedded evaluation helped improve the rollout of the COVID-19 vaccine in New Zealand by identifying challenges and successes early.

Contribution

The paper introduces a novel approach of using an embedded evaluation with appreciative inquiry to support a complex health initiative.

Findings

The embedded evaluation provided real-time feedback, enabling early identification and resolution of challenges.

The approach supported the recognition of successful program features and contributed to the overall success of the immunization rollout.

Using an iterative evaluation method allowed for course corrections and improved equity in vaccine delivery.

Abstract

Background

The introduction and roll-out of the COVID-19 vaccine was one of the most important and complex initiatives undertaken by health sectors worldwide. In New Zealand, the Education, Training and Clinical Support to the health sector of the immunisation programme was undertaken by the Immunisation Advisory Centre (IMAC). This paper describes how an embedded evaluation was planned from the outset, acting as a rigorous "critical friend" to the programme: appreciating and recording successes, identifying challenges and opportunities early, and working with stakeholders to generate innovative solutions (Hindsight to Foresight).

Method

The evaluation research team used an established methodology and evidence base, combining complexity and implementation science with appreciative inquiry (CIS-A), previously applied in immunisation strategy and mental health settings. This approach was taken to acknowledge the complexity of the COVID-19 immunisation context.

Both numerical and narrative data were collected to provide information to the programme for endorsement of what was going well and the early identification of challenges. Methods included analysis of IMAC routinely collected workforce data, help line call data, surveys, focus groups and interviews with vaccinators, vaccinees and key stakeholders. There was also direct observation of vaccination sites and settings.

Results

The embedded evaluation process was sustained throughout the COVID-19 immunisation response. Regular assessment and feedback enabled recognition and support of positive features of activity information about challenges with the opportunity for early course correction.

There were specific areas where external evaluation was especially helpful, the focus on important and concise “Toolkit messages for the day” approach and the focus on equity being two examples.

One of the challenges in using an iterative embedded methodology is that what might have been traditional "end of project" evaluation findings have already been raised and, in many cases, acted upon. Changes have therefore already been made and there may be a perception that an evaluation is adding little to project development.

Conclusions

The evaluation was able to assess the overall success and contribution of the programme to the immunisation rollout. We recommend that using a multi-perspective embedded evaluation approach can be of value in implementing complex health sector initiatives.

Figures