Improving the contribution of mathematical modelling evidence to guidelines and policy: Experiences from tuberculosis

TL;DR

This paper discusses efforts to improve tuberculosis modeling to support better health policies.

Contribution

It highlights collaborative efforts to enhance the quality and transparency of tuberculosis modeling.

Findings

The Tuberculosis Modelling and Analysis Consortium is working with WHO to improve modeling quality.

The goal is to support better decision-making through transparent and accurate models.

Abstract

We read with great interest the recent paper by Lo et al., who argue that there is an urgent need to ensure the quality of modelling evidence used to support international and national guideline development. Here we outline efforts by the Tuberculosis Modelling and Analysis Consortium, together with the World Health Organization Global Task Force on Tuberculosis Impact Measurement, to develop material to improve the quality and transparency of country-level tuberculosis modelling to inform decision-making.

Figures

Flowchart of steps involved in a typical country-level modelling project.Fig. 1

Full text

We read with great interest the recent paper by Lo et al. (2022), who argue that there is an urgent need to ensure the quality of modelling evidence used to support international and national guideline development. Good modelling evidence is difficult to generate, as disease and intervention mechanisms are frequently only partially understood; there may be a need to generalise evidence across settings and into the future, where trial evidence sometimes does not reveal implementation challenges, or only measures short or medium term outcomes. Indeed, the GRADE framework itself may not represent the optimal approach to including economic or modelling evidence in guideline development processes.

Ideally, ensuring good quality modelling evidence should take the form of an iterative process of continued engagement between modellers and guideline developers. Such an approach would allow for a continued improvement to modelling approaches, and subsequently strength of modelling evidence, rather than post-hoc rubber-stamping of evidence as either strong or weak (McQuaid et al., 2021). This need for improved modelling evidence extends beyond the development of international guidelines to the wider ecosystem of modelling to inform infectious disease policy. Much decision-making occurs after this step, for example resource allocation modelling to inform intervention optimization (TB Modelling and Analysis Consortium). Given similarities in the process of using modelling evidence to support decision-making, activities from the wider ecosystem could also be usefully applied to guideline development.

The Tuberculosis (TB) Modelling and Analysis Consortium, together with the World Health Organization (WHO) Global Task Force on TB Impact Measurement, have developed material to improve the quality and transparency in country-level TB modelling to inform decision-making. This material includes guidance for country-level TB modelling (World Health Organization and Modelling, 2018, Menzies et al., 2019) and benchmarking, reporting and review processes (McQuaid et al., 2021). These are focussed on the needs of countries making policy and funding decisions, however the optimal process for supporting guideline developers will likely be similar, as the majority of principles remain the same.

The guidance document describes 10 essential principles for country-level modelling, and associated good practices. These include principles of Relevance, Realism, Appropriateness of model structure, Consideration of all evidence, Validation, Informativeness, Transparency, Timeliness, Country ownership and Iteration. A flowchart, taken from the document, outlines the importance of these principles at each stage of a typical modelling project (see Fig. 1). Meanwhile, the benchmarking, reporting and review process operationalised this guidance through a set of quantitative benchmarks against which model assumptions and results could be compared, as well as a multi-stage review process with standardised reporting templates to provide feedback to modellers during the application and the users of modelling results after completion. This process was piloted in modelling applications in Kenya, Bhutan, Indonesia, Mongolia and Myanmar, where its use prompted important changes in the modelling applications as well as identifying wider issues affecting the production of modelling evidence, such as a lack of empirical evidence and capacity constraints. Elsewhere, the WHO Global Tuberculosis Report recently included the results of modelling estimates (World Health Organization, 2022), which similarly received external review to improve quality and transparency.Fig. 1. Flowchart of steps involved in a typical country-level modelling project.Fig. 1

A critical factor in driving the development of the above guidance and review process was a model comparison exercise for TB, held in 2015 (Houben et al., 2016, Menzies et al., 2016). Such exercises and ensemble modelling, with parallels in other disease areas such as HIV (Eaton et al., 2012), offer an alternative route to improving the robustness and contribution of modelling, highlighting uncertainty in model structure and parameters, and the consequences for the evidence produced. Similar exercises have compared the use of statistical models for subnational estimation of TB disease burden (Alba et al., 2022), or contrasted different approaches to modelling disease dynamics (Ragonnet et al., 2017, Menzies et al., 2018). While not necessarily explicitly aimed at guideline or policy development, these comparison exercises act to identify areas of concern and further strengthen good modelling practices, which should lead to improved modelling evidence and policy.

Ideally such reviews and comparison exercises would be routinely applied, to continuously improve the quality and transparency of modelling for decision-making. However, this requires both buy-in and incentives (in particular funding) for all of the key actors involved, including reviewers, evidence producers (modellers) and evidence consumers (such as the WHO, the Global Fund to Fight AIDS, Tuberculosis and Malaria, and country-level stakeholders). Despite significant advances in TB modelling to support decision making in recent years, a lack of funding to support continued implementation of these approaches remains a key risk to ensuring the quality of modelling evidence to inform guideline and policy development. Ideally this funding, which is comparatively cheap, should be included in modelling budgets for policy work routinely.

We wholeheartedly agree with the conclusions of Lo and colleagues (Lo et al., 2022) that further, sustained work is required to continuously improve and ensure the quality of modelling evidence, and commend their efforts to draw much-needed attention to this issue.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.