Conducting two evidence syntheses in six weeks – experiences with and evaluation of a pilot project

TL;DR

This paper describes a pilot project where two evidence syntheses were completed in six weeks instead of the usual 4-6 months, exploring the process and outcomes.

Contribution

The study introduces and evaluates an intensive six-week process for evidence synthesis, identifying practical lessons and implications for future implementation.

Findings

Researchers found the intensive approach improved focus and productivity, allowing effective management of short-term deliverables.

Machine learning tools and early engagement with commissioners helped address uncertainties and streamline the process.

The process lead role was beneficial but could be improved with clearer responsibilities.

Abstract

Evidence synthesis organisations are trying to meet commissioners’ needs for rapid responses to their evidence synthesis commissions. In this project we piloted an intensive process, working to complete evidence syntheses within six-weeks, rather than the standard lead time of 4–6 months. Our objective was to explore how researchers experience working intensively, identify barriers and facilitators, and determine how a more intensive approach to evidence synthesis could be more systematically introduced in the future.

In a pre-planning phase, an intensive work group was established, and two commissions were selected for this pilot project. The evidence synthesis process was divided into two phases: planning and intensive. The planning phase, involved scheduling the intensive phase, exploring new digital tools, and identifying peer reviewers. The intensive phase encompassed the entire evidence synthesis process. Two review teams were formed, each with a team lead supported by a process lead and leadership contact point. Throughout the project, teams engaged in reflective meetings to evaluate and adjust processes as needed.

During the planning phase, teams identified significant uncertainties regarding scopes, research questions, and inclusion criteria. To address this, they engaged with commissioners earlier than originally planned, clarified these aspects, and prepared protocols. Despite some minor deviations from the original plan, both reviews were completed on schedule, with one team expanding their scope due to the absence of eligible studies. Teams operated flexibly, held regular meetings, and found the process seamless due to fewer interruptions. Machine learning tools facilitated rapid study selection. The process lead role, created to guide and evaluate the project, proved beneficial, providing structure and support, although clearer role delineation with the leadership contact point could have improved efficiency.

Overall, the intensive process fostered focus and productivity, allowing teams to manage short-term deliverables effectively. The researchers preferred working intensively with one evidence synthesis over being involved with many projects at the same time. They felt that time use was more effective, and they were able to complete the tasks in a focused way. However, there are several implications that should be considered before implementing an intensive approach in future evidence syntheses.

The online version contains supplementary material available at 10.1186/s12874-024-02334-y.

Figures