Exploring equity in audit and feedback trials: Secondary analysis of a systematic review

TL;DR

This study examines how audit and feedback trials address health equity, finding that most do not explicitly consider equity in their design or processes.

Contribution

The study provides the first secondary analysis of A&F trials using the PROGRESS-Plus framework to assess equity orientation.

Findings

Only 21% of trials were classified as equity-focused, while 44% were not equity-oriented.

Equity-focused trials increased over time, but explicit equity frameworks were rarely used.

Most trials considered factors like age, gender/sex, and insurance status, but rarely embedded equity in feedback processes.

Abstract

One potential approach to eliminating or reducing health inequities for health systems is audit and feedback (A&F). A&F involves providing measurements of quality indicators to health professionals to support continuous quality improvement, and to increase clinicians’ adherence to clinical practice guidelines. In theory, A&F could help direct efforts toward equity deserving sub-groups (e.g., gender-diverse individuals or those living with low-income) by highlighting factors that may place such sub-groups at higher risk of poor health outcomes. In cases where healthcare professionals can make adjustments to their practice or advocate for mitigating supports or services, A&F – when applied broadly – could help to address some health inequities. However, it is unknown whether and how A&F interventions are currently being used to support equity-oriented quality improvement. In this study, we sought to examine the extent to which trials evaluating A&F interventions address health equity.

We conducted a secondary analysis of randomized controlled trials included in the latest Cochrane systematic review on the effects of A&F on professional practice, which included articles published up to 2020. We used the PROGRESS-Plus framework to consider the extent to which variables related to equity were examined in the trials. Based on extracted data, studies were categorized as not equity-oriented, equity-informed, or equity-focused.

Of the 271 articles included within this analysis, 44% of trials were classified as not equity-oriented (n = 120), 35% as equity-informed (n = 95), and 21% as equity-focused (n = 56). The proportion of equity-focused and informed trials increased over the timeline assessed. Only two articles described an equity-oriented framework approach. Only three articles explicitly reported how equity was embedded in their A&F process by highlighting factors including age, gender/sex, and substance use as part of the patient data presented in their feedback. The PROGRESS-Plus factors most commonly considered in the methods or analysis of the trials were age, insurance status, place of residence, and gender/sex.

A&F trials rarely examine or report the extent to which equity issues inform trial design, A&F processes, analyses, and/or interpretations. Our findings suggest a need for future A&F trials that test explicit approaches to incorporating equity-related interventions to address health equity by helping healthcare professionals, teams, and organizations to be more aware of inappropriate discrepancies in care.

Figures

PRISMA flow diagram.

Line graph depicting the trend of equity-relevant A&F trials, over time.Horizontal axis represents year of publication categories. Vertical axis represents frequency of trials published. Legend: Black/Circle: not equity-oriented, Blue/Triangle: equity-informed, and Red/Square: equity-focused.

Heatmap of Frequency of PROGRESS-Plus factors applied in equity-informed (n = 95) and equity-focused articles (n = 56).*Plus categories include factors other than age and insurance including disability, co-morbidities, and mental health.

Full text

Introduction

When health inequalities are due to avoidable factors (e.g., exposure to unhealthy living or working conditions), are systematically associated with social (dis)advantage, and are deemed unfair or unjust, they are considered health inequities. [1,2] To help understand the root causes of health inequities, scholars have developed frameworks outlining the relationships between health outcomes (e.g., life expectancy, mortality, disability) and the social determinants of health. [3] The social determinants of health include factors that may influence health outcomes based on the daily conditions in which people work and live. [4] The influence of the social determinants of health on health outcomes is complex and can take many forms such as reduced access to care for those living in rural regions, an increased risk of adverse health outcomes due to environmentally poor employment conditions, or stigmatizing patient-provider interactions for those who identify as gender or racially diverse individuals. [1,2] Frameworks suggest that social, economic, and political structures of society give rise to socioeconomic positions which stratify populations, resulting in systemic health inequities. [5] Consequently, it is not surprising that eliminating or reducing health inequities remains a daunting task for health systems and governments worldwide.

A focus on health equity as an essential component of high-quality care is a relatively recent development in health care; quality improvement initiatives, in general, have existed for decades. [6,7] One way to improve quality of care is through regional or national implementation of clinical practice guidelines. [8] These guidelines collate the best evidence for clinical practice and outline factors that clinicians should consider at the point of care to better align their clinical decisions and behaviours with evidence-based practices. [8]

Audit and feedback (A&F) is often used to increase clinicians’ adherence to clinical practice guidelines. [8] A&F involves providing measurements of quality indicators to health professionals, teams, and/or organizations to support continuous quality improvement [9]. A&F can improve quality of care when clinicians actively engage with the intervention and use it to support quality improvement efforts. [9] However, it is unknown whether and how A&F initiatives are currently being used to support equity-oriented quality improvement. If A&F initiatives are considering equity-oriented quality improvement, it is unknown which patient or health professional characteristics A&F developers consider to be relevant to equity or what frameworks are used to understand the relationship between health outcomes, sociodemographic factors, and socioeconomic factors. Further, there is evidence that implementation of clinical practice guidelines may contribute to health inequalities by disproportionately improving the health of those who are relatively health advantaged more than those who are relatively health disadvantaged (e.g., those living with low-income or gender-diverse individuals). [10,11] This makes it more difficult for those who need care the most to access quality care, as described by the inverse care law. [12] As such, it is imperative that A&F researchers examine whether and how their interventions are impacting health equity.

Research question

To examine the extent to which A&F trials address health equity, we will answer the following questions:

What proportion of A&F trials are equity-oriented, that is, have an explicit focus on health inequities?Which equity-oriented patient and/or health professional characteristics are being considered in these trials, and how?What frameworks, theories, or conceptual approaches are being used in these trials to understand and address health inequities?

Methods

We conducted this study in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) reporting guidelines and specifically followed the PRISMA-Equity Checklist (S1 Table). [13]

Study selection criteria

We conducted a secondary analysis of articles included in the latest update of the Cochrane systematic review on the effects of A&F on professional practice [9]. For that systematic review, studies were identified from the following databases: MEDLINE (Ovid), EMBASE (Ovid), CINAHL (EBSCO), the Cochrane Library (CENTRAL), clinicaltrials.gov, and WHO ICTRP (searched to February 2019 due to COVID-19 pandemic). The complete search strategy for that systematic review is available elsewhere and the study selection criteria are outlined in Table 1, with the exception of “intervention & domain” which reflects the domain of our secondary analysis, not the systematic review [9]. For this secondary analysis, we included all studies that were selected for the systematic review. In cases where an article reported on multiple primary outcomes, we considered the article for equity-orientation only once, as a whole, not for each outcome separately.

Table 1: Study Selection Criteria.

Data extraction & analysis

A pre-determined data collection form was used to collect information on study design, participants, setting, interventions, outcomes, and results for the Cochrane review [9]. For this study, additional data extraction related to equity-oriented data was collected using a Microsoft Forms data collection form. The form was tested by four reviewers with three articles and modified as needed before extracting all included studies. Included studies were independently extracted by two reviewers in duplicate. Disagreements and discrepancies were resolved through discussion and/or by a third reviewer, as needed.

Data extraction included study characteristics, intervention descriptions, recruitment processes, findings, discussions related to each PROGRESS-Plus factor, and frameworks, theories, or conceptual approaches used to address health inequities. The PROGRESS-Plus framework is a comprehensive listing of such socially stratifying factors that may be related to inequities (S2 Table). [14] PROGRESS stands for place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital. [14] “Plus” factors include other personal characteristics that may attract discrimination (e.g., age, disability, substance use), features of relationships (e.g., smoking parents), time-dependent relationships (e.g., leaving hospital) and any additional context-specific factors that may indicate disadvantage. [15] The PROGRESS-Plus framework was developed by the Campbell and Cochrane Equity Methods Group as a tool to systematically identify and analyze health disparities in public health and policy research. [14–16]

Data synthesis

The complete data extraction form is available in the supplemental information (S1 File). Following a similar process used by Lu et al., [17] reviewers identified each study’s equity-oriented classification based on extracted data as one of three categories:

a)Not equity-oriented: Study does not have an explicit equity-oriented objective AND no PROGRESS-Plus factors are used in the data analysis/results as stratifying variablesb)Equity-informed: PROGRESS-Plus factors are used in the data analysis/results as stratifying variables BUT the primary outcome is not reported for a PROGRESS-Plus defined groupc)Equity-focused: The study objective explicitly pertains to equity AND the entire study population is defined by a PROGRESS-Plus factor or a PROGRESS-Plus factor is used to stratify reporting of the primary outcome

This classification system emphasizes that only reporting on PROGRESS-plus factors in a study is not sufficient for it to be equity-oriented (for example, although most studies regularly report on participant sex, most are not designed or expected to impact gender/sex equity). We defined an equity-oriented study as one that either intentionally uses PROGRESS-Plus factors to better understand their data (equity-informed) or sets out to better understand or impact an explicitly identified health inequity (equity-focused).

To better understand the current approaches to incorporating health equity into A&F trials, we completed a sub-analysis for articles that were categorized as equity-focused. For these articles, extracted intervention data was further analyzed to determine if and how equity was considered in the A&F process of the trial intervention. Specifically, we looked at whether PROGRESS-Plus factors (different from the A&F trial’s outcome or population of focus) were explicitly identified in the ‘Audit’ or ‘Feedback’ portions of the A&F trial.

Descriptive statistics were used to organize our data by equity-oriented classification and PROGRESS-Plus factors.

Results

Equity oriented classification of included articles

Described in detail elsewhere, the updated systematic review retrieved 20,171 articles, of which, 276 were included in the systematic review [9]. From these, 271 articles were included in this secondary analysis. The five excluded articles had an identical author, study design and intervention as an already included article, but with a different primary outcome of focus. Because of the scope of this secondary analysis, in cases where identical articles reported on multiple primary outcomes, we considered the articles for equity-orientation only once, as a whole, not for each outcome separately. A detailed list of general characteristics of included studies is presented elsewhere [9]. From the 271 articles included within this analysis, 44% of trials were classified as not equity-oriented (n = 120) and 56% as equity-oriented (n = 151). Within equity-oriented trials, 35% were classified as equity-informed (n = 95) and 21% as equity-focused (n = 56) (Fig 1). Fig 2 shows the trend of trial classifications over time, with the earliest included study from 1980 to the latest in 2020. The number of equity-focused and equity-informed trials has increased over time, with the greatest proportional increase seen in equity-focused A&F trials, with the majority (n = 32, 57%) being published after 2010.

PRISMA flow diagram.

Line graph depicting the trend of equity-relevant A&F trials, over time.Horizontal axis represents year of publication categories. Vertical axis represents frequency of trials published. Legend: Black/Circle: not equity-oriented, Blue/Triangle: equity-informed, and Red/Square: equity-focused.

Equity oriented characteristics reported within included articles

As expected, given our classification criteria, PROGRESS-Plus factors are reported more frequently throughout article sections in equity-informed and equity-focused articles compared to articles that were not equity-oriented (Table 2). Only two equity-focused articles described an equity-oriented framework approach: the chronic care model [20], and the PROGRESS-Plus equity framework [29]. (Table 2)

Table 2: Overview of Reported Equity-Oriented Characteristics by Article Section.

Frequency of specific PROGRESS-Plus factors applied in equity-informed and equity-focused article sections are illustrated in Fig 3. The Plus factors, including age and insurance, were present across both equity-informed and equity-focused articles. This was followed by place of residence (e.g., rurality) and gender/sex. Religion was reported the least (5% in equity-focused). (Fig 3)

*Heatmap of Frequency of PROGRESS-Plus factors applied in equity-informed (n = 95) and equity-focused articles (n = 56).Plus categories include factors other than age and insurance including disability, co-morbidities, and mental health.

Upon further analysis of the 56 equity-focused articles, we identified three articles that explicitly reported how equity was considered in their A&F process (Table 3). Two articles highlighted PROGRESS-Plus factors as part of patient data in their feedback or organized patient data based on factors including substance use (plus), age (plus), and gender/sex. [30,31] Based on O’Neill’s definitions of the PROGRESS-Plus factors, gender/sex and personal characteristics including substance use and age may limit a person's ability to manage their health or obtain healthcare and may be associated with stigmatization and discriminatory behaviours. Additionally, gender/sex may indicate differential access to healthcare services or health risks [14] One study broadly stated that their feedback included a discussion about “structural, organizational, and social barriers to change” which could be inclusive of any or all of the PROGRESS-Plus factors. [32] (Table 3)

Table 3: Sub-Analysis of Equity Embedded in Feedback within Equity-Focused Articles.

Discussion

We found that the number of equity-focused trials evaluating A&F has increased over the last three decades, despite representing a minority of A&F trials, overall. In our review, only two trials used an equity-oriented framework in their approach (3.6%), and only three studies explicitly reported how equity was embedded in their A&F process (5.5%).

Our approach follows the secondary analysis of diabetes quality improvement trials reported by Lu et al. in 2018. [17] However, they identified double the number of equity-focused trials compared to equity-informed, whereas we saw the opposite (Fig 1). Additionally, our study cut across multiple health issues, whereas Lu et al. focused solely on one health condition. [17] The inclusion of multiple health issues could contribute to this difference in our findings. Despite these differences, there are several similarities between our findings. We found an increase in equity-oriented studies over time (Fig 2), and that age and gender/sex were the most cited PROGRESS-Plus factors (Fig 3), which were similarly reported by Lu and colleagues. [17] Furthermore, we found that 36% of equity-focused trials used PROGRESS-Plus factors to stratify outcomes (Table 2) – higher than the 23% identified by Lu et al. [17]

We acknowledge some subjectivity in how we identified PROGRESS-Plus factors. For example, we debated whether nursing home populations should be considered within the age (plus) factor and ultimately decided to include them if the population was described as being at a particular disadvantage (e.g., an institutionalized care setting). Relatedly, the process of categorizing trials as ‘not equity-oriented’, ‘equity-informed’, or ‘equity-focused’ was complex. It was sometimes difficult to identify equity-orientation as a goal of the intervention versus as a feature of or confounded with the clinical topic of interest. We developed a set of criteria, as described in our methods, for selecting a category, yet there were times when each reviewer chose different categories, and the final decision required team discussion. For example, an article may have used a PROGRESS-Plus factor to stratify reported outcomes which, early in the process, would have led us to classify it as equity-focused. However, in some circumstances, there was a lack of intentionality or context for the stratification of outcomes, and this led to a classification of equity-informed instead. These differences and discussions led to some refinement of the criteria over time. Although we reviewed all classifications at the end of the process, it is possible that the evolution of the criteria and our thinking over time resulted in some discrepancies in classification. Furthermore, we did not assess differences in efficacy between equity-oriented and not-equity oriented trials. However, such an analysis would be extremely complex and at high risk of bias due to the variability within equity-oriented studies regarding intervention components, equity concerns, and populations.

Potential impact & future directions

Our findings suggest an opportunity for A&F trial developers to improve how they embed and report on equity. Trials that aim to be more inclusive can follow trends within the equity-focused trials found in this study, including explicitly identifying equity-related factors within study objectives, considering how factors like the social determinants of health may provide a health-related (dis)advantage and affect study outcomes within study backgrounds, and conducting sub-group analyses for distinct equity-related factors that may be relevant to the population of interest. We identified only four trials that explicitly reported feedback that was inclusive of equity considerations extending beyond the trials’ population of focus. There is a significant opportunity for A&F intervention designers to create formal pathways to embed equity within feedback, to facilitate better health outcomes for disadvantaged populations. This secondary analysis was a first step in exploring if A&F interventions were considering equity and how; we hope this work encourages others to do additional analyses to further outline differences in efficacy between trials that are equity-oriented and those that are not.

Conclusion

Despite the substantial evidence base in support of A&F to improve care in general, there remains limited evidence on whether (or how) to use A&F to address health inequities. This study reviewed A&F trials to examine the extent to which trials consider health equity over time and provides examples of ways to embed equity in implementation trial designs, A&F processes, and reporting that may be replicated in future trials.

Supporting information

S1 TablePRISMA-Equity Checklist.(DOCX)

S2 TableDefinition of PROGRESS-Plus Factors.Adapted from Lu et al. (2018) Supplemental info.(DOCX)

S1 FileData Extraction Form.(DOCX)