Evaluation of a Participatory Action Project to Address Opioid Misuse: Breaking Down Barriers Through Partnership Processes

TL;DR

This paper evaluates a community project to tackle opioid misuse by analyzing partnership processes and their impact on public health outcomes.

Contribution

The study introduces a mixed-methods evaluation approach combining the CBPR model and Ripple Effects Mapping to assess community partnerships in public health.

Findings

Trust, equitable relationships, and structured methods were key to successful partnerships.

Ripple Effects Mapping revealed broader community impacts beyond initial project goals.

Mixed methods confirmed the importance of community context and participatory processes in public health projects.

Abstract

Community based participatory research and participatory action research are increasingly being used to engage communities in addressing social and health disparities. There is a need to develop broadly applicable evaluation methods that can be used across participatory project environments to identify the processes critical for addressing complex public health issues, as well as the productiveness of community research partnerships. We present a case study of a community participatory project conducted over three years and our evaluation approach. We used the Community Based Participatory Research Conceptual Model as the framework for the evaluation surveys (n=9) and interviews (n=7) with project participants, querying perspectives on the four model domains: community context, partnership processes, intervention and research and outcomes. In addition, we conducted a Ripple Effects Mapping (REM) exercise with ten community members to determine the broader impacts of the project on the community. This mixed-methods approach permitted us to confirm findings from quantitative surveys with qualitative findings from interviews and the REM. Key processes identified as facilitators to a productive partnership and positive outcomes include a context of trust, effective implementation of processes that establish equitable partner relationships and partnership synergy, a clearly defined focus for the partnership and a structured participatory research method that helped break down silos and mobilise the community for action. Our project evaluation approach, combining the CBPR model and REM, guided measurement of common metrics that are key to effective community engagement as well as exploration of unanticipated outcomes.

Figures

Full text

Introduction

Community based participatory research (CBPR) and participatory action research (PAR) are now widely accepted as legitimate methods for addressing social and health inequalities in marginalised communities (McElfish et al. 2015; Wallerstein & Duran 2006). In these approaches, researchers and community representatives work together to build on the strengths of the community, sharing responsibility for the research process. CBPR and PAR have a different epistemological foundation than other research methods (Meyer 2000). They seek to integrate action, theory, practice and critical self-reflection in the pursuit of pragmatic solutions to community concerns (Jagosh et al. 2012; Reason & Bradbury 2001). As a way to address public health concerns, the collaboration between academics and community members focuses on enhancing understanding of a health concern, as well as implementing tailored interventions and policy changes based on what has been learned (Israel et al. 2010).

Despite the increased use of participatory research, there are fewer available standardised evaluation measures and methods for CBPR and PAR compared to conventional research approaches. Several recent reviews have identified measures to evaluate the quality and outcomes of community engaged research collaborations (Luger, Hamilton & True 2020; Tigges et al. 2019). With the exception of a few cases, such as the work of Oetzel et al. (2015), the majority of measures have not been thoroughly examined or tested (Israel et al. 2020; Sandoval et al. 2012). The challenge in evaluating participatory research stems from identifying which aspects of engagement are critical for achieving the desired outcomes (Oetzel et al. 2018). Furthermore, it can be difficult to compare participatory research interventions because evaluation approaches tend to be tailored to the specific context (Meyer 2000). Without validated tools and models, it is difficult to apply evaluation methods across different contexts.

Evaluating the productiveness and structural arrangements of research partnerships is another challenge (Belone et al. 2016). Multidisciplinary processes are difficult to manage and their outcomes are often unclear or subject to interpretation. Important process indicators, such as relationship quality, trust, participation and relational dynamics, can be hard to measure objectively (Höppner 2009; Pohl & Hadorn 2008), yet they are critical because they drive the productivity of the partnership and are important outcomes in their own right. Participatory research practitioners must also reflect on the challenges faced when conducting and evaluating research to improve their understanding of best practices (Dedding et al. 2021) and disentangle the multiple ways that engagement contributes to achieving project aims. Case studies can enhance the knowledge base for participatory research evaluation approaches by illustrating the use of specific frameworks, methods and constructs (Hicks et al. 2012; Reese et al. 2019; Sandoval et al. 2012).

This paper seeks to advance the scientific understanding of participatory research partnerships by highlighting our evaluation results, following a description of the project and its outcomes. We demonstrate the complexities of evaluating participatory research by presenting a case study focusing on the evaluation methods and framework that were employed.

Background

Opioid misuse is a significant health issue globally. In response to the dramatic increase in opioid dependence and overdose deaths in the United States, the U.S. Department of Health and Human Services (2017) declared the opioid crisis a public health emergency on 26 October 2017. There is substantial variation in community rates of opioid misuse and overdose, and the availability of prevention and treatment services, necessitating community designed solutions in addition to national action. We report on the evaluation of a participatory action project in a rural Virginia community that wanted to further its efforts to tackle the opioid crisis.

Virginia has been greatly impacted by opioid misuse and overdose. The rural community that is the focus of this project, Martinsville City/Henry County, had one of the highest average per capita opioid prescription rates in the U.S. between 2006 and 2012 (U.S. News & World Reports 2017). As reported previously, the community’s opioid overdose death rates were three times higher than the state average in 2016, and the area had the highest rate of opioid overdose emergency room visits statewide in 2018 (Virginia Department of Health n.d.; Zimmerman et al. 2020b). Much like other rural communities, this county had lower levels of educational attainment, higher levels of disability and lower median household incomes (United States Census Bureau 2020). Once a prosperous farming and manufacturing center, the community experienced economic downturn as factories closed through the 1980s, 90s and 2000s.

Despite the significant challenges facing this resilient community, it mobilised to form an opioid task force in 2016. The task force was composed of key community organizations including the police department, behavioral health providers, peer and faith-based recovery programs, the local hospital and representatives from the judicial system. To formalise a process to take action, the task force aligned with an existing community-academic research partnership, Engaging Martinsville (EM). EM decided to use a community-engaged approach, the Stakeholder Engagement in Research Question Development and Prioritization Method (SEED Method), that they had previously used in the community to address a disparity in lung cancer mortality (Rafie et al. 2019). With support from the opioid task force, EM implemented a project using the SEED Method to develop community action plans to address the opioid problem (Zimmerman et al. 2020b).

Involving residents and local stakeholders in action planning is important to ensure that a community’s strategies are based on an understanding of the local situation and reflect the community’s concerns and needs (Kapiriri & Norheim 2002; Lomas et al. 2003). Without community buy-in, many potentially effective strategies, such as harm reduction, may not be achievable (Childs et al. 2021). Currently, there is a mix of top-level leadership approaches, coalition approaches and evidence-based approaches being offered as options for engaging communities. While coalition approaches generally involve a variety of community stakeholders, they may not be founded on participatory values (such as inclusion of people with lived experience) and are not easily transferable to other communities (Windsor 2013).

The SEED Method was designed in response to a need for systematic, participatory engagement methods that engage people with lived experience to create research agendas or action plans. The SEED Method addresses the limitations of effective community stakeholder engagement, including the need for technical training of stakeholders and community capacity building (Hoffman et al. 2010; O’Haire et al. 2011; Oliver et al. 2004), by incorporating research training and providing facilitation tools that lead the teams through the process of stakeholder selection, conceptual modeling and strategy development and prioritization. Participants in past SEED projects indicated that SEED prepared them well for the tasks they were asked to perform and that they had a sense of satisfaction at gaining new skills in the process (Rafie et al. 2019; Zimmerman et al. 2017). Results from previous SEED Method demonstration projects strengthen the evidence that engaging diverse groups of stakeholders in research results in pinpointing more comprehensive priorities for health research, and that SEED has application in identifying and prioritizing actions to address a variety of community health challenges. We anticipated that applying the SEED Method as a participatory action planning tool within this community would result in collaborative actions on community-derived strategies and help strengthen partnerships.

The SEED Method combines collaborative, participatory and consultative roles to engage community members in identifying and prioritizing strategies. The method engages stakeholders at three levels: (1) Community Research Team (CBPR Team) – a multidisciplinary partnership that collaboratively leads the project; (2) Topic Groups (TGs) – participatory stakeholders recruited based on their experience and knowledge of the health topic; and (3) SCAN participants – consulting stakeholders who participate in focus groups and interviews. At each level, stakeholders make unique contributions to the project but work together iteratively (Zimmerman et al. 2020a). We modified the SEED Method, originally developed to focus on research question development, to include facilitated steps for action planning and intervention development. Figure 1 outlines steps of the modified SEED Method.

PARTICIPANT INVOLVEMENT

The project involved a total of 85 community stakeholders at various levels of engagement. Eight community members formed part of the initial participatory community research team, EM. Over the course of the three-year project, five EM team members left due to other obligations and family health issues. At the same time, three new members joined the team to replace those who had left. EM team members included three peer substance misuse counselors, one nurse, two prevention specialists, two medical administrators, two university faculty members and a graduate student. The EM team met weekly at the beginning of the project, decreasing to fortnightly at the end of the first year. At the beginning of the project, most meetings were held in person at a local community facility. In response to the COVID-19 pandemic, team meetings shifted to being fully virtual. The shift in meeting format did not negatively impact the team’s work; in fact, it allowed for more consistent attendance. The community members of the research team were paid an hourly minimum wage for all project-related work.

The EM team chose three stakeholder groups to form the Topic Groups. The team recruited 21 community members for TGs composed of: (1) family and friends impacted by opioid use disorder; (2) service providers (e.g., police, EMT’s); and (3) healthcare providers. TG members were recruited through newspaper advertisements and fliers posted and distributed through email to key organizations. Compared to the community demographics, EM and TG members were slightly overrepresented by individuals of White race with a higher level of education. Their ages ranged between 21 and 64 years, 65 per cent reported having a college degree and 14 per cent identified their race as Black. Each TG met seven times and meetings were facilitated by members of the EM team. Meetings took place at a local healthcare facility. Participants received a stipend ($250) for their participation and food was provided during meetings.

In addition to research teams and TGs, community members were invited to participate in focus groups. Twenty-four community stakeholders participated in four focus groups and provided additional community perspectives on the opioid issue. The TGs and EM team each chose unique stakeholder groups from whom they wanted to obtain information to inform their work. Participants were recruited by the EM team through multimedia advertising. EM conducted the focus groups and summarised the information for presentation to the TGs. Focus group participants received a stipend ($25).

Following the creation and prioritization of strategies by the TGs, they were presented at two community meetings attended by 41 community members. During the meetings, community stakeholders voted on a short list of strategies. The three-hour meetings took place approximately four weeks apart at a local community facility and dinner was provided. The attendees chose four strategies to work on over the next year and formed a work group for each strategy.

A total of 29 stakeholders joined the four working groups that were formed at these community meetings. Some of the stakeholders had participated previously (e.g., as TG members) and some were new to the project. The working groups created their own meeting schedules, but generally met monthly. The working groups met for 12 to 24 months and received continuing support from and communication with the EM team. Initially, most meetings occurred in person, but later transitioned to virtual or other formats in response to the COVID-19 pandemic. Despite this change of meeting format, the teams continued to work effectively together and accomplished their objectives. The EM team arranged quarterly meetings with the work group coordinators to discuss progress and challenges. Bi-annually, EM hosted meetings bringing together all four work groups to update each other on progress and share a meal. Work group members were not compensated for their work on the strategies.

Evaluation Approach and Methods

We used the CBPR Conceptual Model (UNM College of Population Health n.d.) as the framework to evaluate the effect of the SEED Method on partnership processes and their impact on partnership synergy and project outcomes. This model was developed by Wallerstein and colleagues through a two-year pilot study that looked at how the CBPR process influences or predicts outcomes (Wallerstein et al. 2008; Wallerstein & Duran 2010). The model addresses four CBPR domains (contexts, partnership processes, intervention and research and outcomes) and outlines the potential relationships between each. The first domain consists of contextual factors that shape the nature of the research and the partnership, such as social, structural and political factors. The second domain, partnership processes, includes the partnership structure, members and member relationships, which interact with contextual factors to shape both the intervention and its research design. If partnering practices are effective, they will result in mutual learning, an ability to work together effectively and partner synergy, which impacts outcomes. The third domain consists of the intervention and research processes aligned with CBPR principles. These factors, taken together, result in intermediate system and capacity changes, and ultimately health outcomes (Oetzel et al. 2018). The evaluation tools associated with the CBPR model were validated initially by Oetzel et al. (2015). More recently, Boursaw et al. (2021) evaluated the psychometric properties of seven revised scales that corresponded to the four CBPR model domains. We modified the survey and interview guides developed for the purpose of evaluating constructs within each of these domains (Engage for Equity n.d.; UNM College of Population Health n.d.). The project was approved by the Virginia Tech Institutional Review Board (VT IRB 18–860) and all individuals participating in research procedures provided signed informed consent.

EVALUATION QUESTIONS AND ANALYSIS

The community research team selected constructs within each of the CPBR model domains that they felt were most relevant to their partnership and selected questions from the survey that focused on these constructs. Within the context domain, we evaluated the social and structural context of our community and our partnership capacity. In the process domain, we focused on partnership structure including the ability to bridge differences, the partnership core values and alignment with community engagement principles. We also included constructs related to relationships, including quality of dialogue, participatory decision making, leadership, resource use and trust. We were particularly interested in evaluating community involvement in research in the intervention and research domain, which has been shown to impact outcomes. Within the outcome domain, we focused on community organization outcomes and partner challenges, health outcomes and project sustainability. In addition, we included two questions about the overall quality of the partnership work and satisfaction with the partnering experience. The final survey consisted of 64 questions that focused on 15 constructs within the four CBPR domains (See Appendix 1). All survey questions were on a Likert scale of either six or seven points, which varied depending on the construct. We calculated the mean score for each construct in a domain. Mean scores of four were considered indicative of a positive response for six-point Likert scales and a mean score of five for seven-point Likert scale constructs. We used the scores to evaluate the effectiveness of the partnership and to inform conversations with the community research team about partnership sustainability and future activities.

The survey was developed for online access using Qualtrics. Project collaborators (members of the research team, TGs and community work groups) were invited to complete the survey through an email invitation approximately 28 months after the start of the project, which had a duration of 36 months. Nine TG members participated in the survey.

QUALITATIVE INTERVIEWS AND ANALYSIS

Current and former members of the research team were invited to participate in qualitative, one-on-one interviews using a modification of the CBPR Conceptual Model interview guide (Engage for Equity n.d.; See Appendix 2). Seven team members participated in the interviews.

A research assistant who was not associated with the project conducted the interviews. The semi-structured interviews lasted approximately 30 minutes and took place on Zoom. The interview guide queried constructs within each of the four CBPR domains to mirror those included in the survey, and solicited recommendations for improved partnership management and dissemination of project findings. Interviews were recorded and automatically transcribed by Otter.ai, a third-party vendor. Transcriptions were checked for accuracy prior to coding. All coding was conducted using Taguette (version 0.10.1), a third-party vendor, and computer-assisted qualitative data analysis software. We used the framework method for transcript analysis (Gale et al. 2013). Initial coding was conducted by the research assistant under the supervision of a member of the research team. Transcript analysis began with a focus on trust within the partnership, facilitators and barriers to trust, and the impact of trust on the effectiveness of the SEED Method in community action planning.

The research team expanded on the preliminary analysis using the CBPR model domains and constructs as a framework for coding and theme development. An initial code list was created deductively for each domain of the CBPR model, incorporating the codes related to trust developed from the initial analysis. Upon review of each transcript, additional codes were added inductively as appropriate. The finalised codes were grouped by domain and constructs within each domain. They were then categorised into broader themes under each construct. The recommendations made in response to the final interview question, ‘If another group were going to start this kind of partnership, what would you tell them in order to help them be successful?’, were grouped and evaluated for common themes.

ADDITIONAL EVALUATION METRICS

Several additional assessment tools were used to evaluate the process and outcomes of the study. Demographic information was collected from project collaborators. In addition, EM and TG members were asked to complete a group readiness questionnaire used in previous SEED projects early in their work together. Other data useful for the evaluation included community perspectives on the opioid crisis gathered through focus groups (Hargrove et al. 2022), the strategies developed and prioritised by the TGs and selected for action by community stakeholders, and work group accomplishments, the details of which will be reported elsewhere.

We also conducted a Ripple Effects Mapping exercise with project partners to capture information about both intended and unintended consequences of the project (Chazdon et al. 2017; Emery et al. 2015). We followed the process described by Chazdon et al. (2017). The REM process visually maps intended and unintended impacts of a project and employs four key elements: (1) appreciative inquiry; (2) a participatory approach; (3) interactive group interviewing and reflection; and (4) mind mapping that is used to chart the chain of effects of the project.

Appreciative inquiry is a process of asking unconditional, positive questions that strengthen a system’s positive potential. In the context of REM, appreciative inquiry is used to facilitate the discovery of participants’ best experiences with the program or project, and the unanticipated effects of the project in the community (ripple effects). We employed a participatory approach to the conduct of the REM, involving the research team in the choice of questions asked during the REM, the conduct of the exercise and analysis of the outcomes.

The REM exercise was conducted over a two-hour Zoom meeting. The exercise was facilitated by a member of the research team, assisted by members of EM. We used the online mapping program, Mind Meister [2023], to illustrate the participants’ best experiences and community impacts as they were discussed. We invited diverse participants, including members of the research team, TGs and work groups, as well as focus group and community meeting participants. Ten community members participated in the session.

The exercise began with a brief overview of the REM process, followed by a 15-minute period in which participants interviewed each other. They were provided with specific instructions on how to conduct the interviews. They selected three questions from a longer list of questions developed by the research team. Participants were paired up and directed to Zoom breakout rooms for this activity. After the interview session, participants shared and discussed the most salient points from their interviews. The facilitator used the Mind Meister program to lead the attendees in a process of visualizing the project impacts and illustrating cause-and-effect relationships between them (mind mapping). Using a rippling and theming approach, topics were displayed in the map and similar topics were positioned together under common themes. Participants commented on what they thought came about and any attributions to the causes. A final 15-minute group meeting allowed participants to reflect on how the mapping process made them feel. They also reflected on what was interesting about the map and the story it told.

Following the REM session, minutes from the meeting were reviewed, and information from the map was downloaded in outline format and edited for clarity. Two members of the research team reviewed the notes from the meeting and the map contents and identified themes and REM outcome areas that were informed by the REM guiding questions. Themes and information from the REM exercise were then organised according to the level of impact (i.e., individual, interpersonal/community, environmental).

Results

Key themes from the survey (Tables 1, 2, and 3) and interviews (Figure 2) included the importance of building relationships and trust to the community partnership and partnership synergy, as well as the impact of the structured method of engagement (the SEED Method).

BUILDING RELATIONSHIPS

Processes for quality and equitable relationships among partners are important for sustainable, productive community partnerships. Factors that promote good partner relationships include quality dialogue, participatory decision making and strong leadership. Survey results show a strong mean score for quality dialogue (5.54/7, s.d. 1.21) among the project partners, which was composed of questions related to mutual respect, good listening and conflict resolution. The project also had strong, effective leadership as reflected in the mean score for this construct (5.1/6, s.d. 0.37), which is associated with team synergy (Table 1).

In interview responses, relationship building is seen as an outcome of treating each other with mutual respect and engaging in participatory decision making.

For instance, when we began our team, we started out by first of all getting to know each other, trying to create an environment where we got to know each other on an equal basis.But we tried to create an environment where we’re all equal partners in this and that decisions are made equally.We use first names, we share decision making, and we consider everybody around the table to be an expert in whatever it is that they kind of represent. So, if it’s a community member, they may be an expert.And once everybody kind of made their argument, a lot of times at that point we could come to an agreement.So, it’s kind of a combination of getting to know the individuals as persons, but also working together to identify what’s going on in the community, what the needs are and what the interest is in addressing those.

Strong leadership, community inclusion throughout the project, and community members serving as leaders and decision-makers strengthened the partnership.

I don’t have the degrees that other people have, the educational degrees. I’m still made a part of the group. I think that the partners still trust my suggestions and judgments and have made me a part of the whole process.I think [NAME 1] and [NAME 2] have exemplary leadership, when it comes to some of these situations where like say going into a community that may have some resistance.I feel that my voice is heard in the meetings. I’ve even helped bring in some other people that I would think that had a lot to offer to this issue, and they are very much welcomed into the group and their voices being heard.

TRUST

Trust is integral to collaborative projects and provides a foundation for project sustainability and successful outcomes. The survey findings indicate a significant level of trust between those working on the project (mean = 6.29/7, s.d. 0.19) (Table 2). This is significant, as trust and team synergy are associated with project sustainability and outcomes. These positive partnership process scores may be a consequence of the partnership structure that incorporated community engagement principles resulting in strong relationships built on trust.

The importance of trust to the partnership was a repeated theme in the interviews, as well. This included trust between individuals, academic and community partners, and trust in the process. Familiarity, purposeful mutual understanding, being present over time, and ‘doing what you say’ were factors that built trust. Other factors considered important were having a consistent presence in the community throughout the project, especially during challenging times, and the commitment of the partners to each other and to the goals of the project.

So, they, they had to trust us. And yet, we also had to trust them. Because it’s not going to work if it’s just people from the university trying to make something happen. And so, we have to trust them also, that they’re going to put in their part.So, it’s kind of like this mutual trust, kind of expectations of one another. And then we’re going to see whether or not those actually get fulfilled.I think that we built a level of at least trust and support for one another, they can come to us, and we can come to them.And as long as our team continues to be safe, do what we say, right, and be consistent in our presence, we’re going to do what’s on the part of everybody, right?One thing is you have to trust the structure itself. Like, if you don’t think it’s a good process, then the structure probably isn’t going to help.

Conversely, turnover of team members that can occur with lengthy projects was identified as a hindrance to trust.

But sometimes people just stop participating. And you don’t know why. And I think that kind of challenges trust, even throughout the team.

The history of previous work of the Engaging Martinsville team, and the reputation of the two universities involved in the project, were credited with the community overcoming a cultural aversion to outsiders and skepticism about their motives. In fact, the community reached out to the EM team for assistance in addressing the opioid issue and breaking down existing social barriers to action. The community had a good deal of trust that the EM team had their best interests in mind.

We had a great community presence there because we had done a community needs assessment already with a local coordinator and she continued working with us as the coordinator with the lung cancer project that had some wonderful outcomes. Then the community came back to us and said you know we’ve got this opioid problem, and we’d like to apply SEED to that so then we just kept going so it’s been a while.So over time, with being more personable, and understanding each other trust has grown, to the point where I don’t think that anything on the academic side is ill-willed or malicious, and I don’t think on the community side either.And, because we’ve been doing this for so long, the trust is now based on experience versus based on some assumptions.

USING A STRUCTURED METHOD OF ENGAGEMENT

Team members credited the partnership processes that are part of the SEED Method with helping to develop trust and allowing them to feel safe to express themselves without judgment, put aside their own biases, and work together. They felt the SEED process provided the safety and structure to build trust between individuals and institutions.

Although the partners came from varying backgrounds, responses to the survey showed that the process was able to bridge that gap to unify the partners around a core understanding and vision for the project (core values mean score = 6.35/7, s.d. 0.21 – Table 3). This may have been accomplished through the effective implementation of the principles of community engagement that are part of the SEED Method as indicated by the survey findings of significant alignment with principles of community engagement (4.82/6, s.d. 0.21) (Table 3).

Interviewees highlighted the importance of a structured process that moved partners from research to action and credited the process with breaking down barriers to collaboration to accomplish goals. They identified the accomplishments of the four work groups on the prioritised actions as the most important outcomes of the project, and credited the process for getting the community working together. Less tangible outcomes included the impact of the project on networking within the community, and the capacity of the community to approach a problem and set achievable goals.

Reminds me of like a mediator, not that they were mediating. But you know, it was just having that third party with a structure and a framework, and a goal, and very specific tasks to keep us on track, you know, track and focus.They have been trying to get this [community strategy] done. Why did it happen? I think it was just a result of going through this process and getting enough people sitting at the table and having the community say we want this and we’re going to support it, that then was able to drive it forward.We still work for the [public agency], but it’s amazing, we have a partnership with a private hospital, and we are a public community service. This has never happened!We are not the silos to the extent that we were, I really don’t think so. I think that’s the greatest success (of the project), that we acknowledge that we need each other.

The team also attributed the breakdown of interpersonal barriers between team members and other stakeholders to the CBPR principles of shared power, mutual respect and shared decision making that are foundational to the SEED Method.

I think the SEED method is just a great best practice as far as trying to eliminate from one person dominating the outcomes, because the whole process is used to weed that out, because everyone has to participate.I was astounded because [NAME 2] and [NAME 3] and the way the SEED method is set up, provided this platform that was safe, without interference of personal bias… they did just such a great job keeping things neutral. And when that environment was established, then the partnership of trust could really blossom.

OUTCOMES FROM THE RIPPLE EFFECTS MAPPING

In the Ripple Effects Mapping exercise, participants described project outcomes that increased community awareness, improved collaboration between community organizations, particularly the physical and mental healthcare organizations, and increased the availability of local services (Figure 3). They described how these outcomes resulted in reduced stigma, increased community connectedness and understanding, and reduced isolation for those impacted by substance misuse. Networking and collaboration improved between community organizations, particularly the physical and mental healthcare organizations, resulting in systems for ensuring a continuum of care for individuals with substance use disorder (SUD), expansion of peer recovery programs and access to medically assisted treatment (MAT). Team members credited the systematic approach of the SEED Method with fostering this collaboration through open dialogue, coordination of effort and sustained engagement (Figure 3).

Finally, presentations by the team increased awareness of key agencies that could support local responses to socioeconomic, cultural and environmental conditions. Federal and state funds supported the establishment of a new recovery and residential center, a regional recovery court and prevention education that served the region. Networking across county lines increased the readiness of adjacent counties to address the opioid and substance misuse issue and afforded the opportunity to bring services to those communities.

Discussion

Evaluation of community engagement is a topic of much research, as it is essential to bridge the gap between science and practice. Models have been created to define meaningful engagement and assess its impact (Khodjakov et al. 2011; Organizing Committee 2022; Wallerstein et al. 2008). These models define the key domains of community engagement principles with indicators of impact when applied effectively (Organizing Committee 2022). The diversity of context in which community engagement occurs has necessitated that these models be flexible and provide a range of outcome options. Specific common metrics that can be used across community-engaged research projects are still needed.

This project serves as a case study for evaluating community engagement practices. Using the CBPR model and tools, we were able to evaluate the relative success of our community-engaged project in relationship building and developing trust within our community-university partnership, the ability to create significant changes benefitting the community, and the impact of the SEED Method on these outcomes. By evaluating the various domains within the CBPR model (i.e., context, partnership processes, intervention and research and outcomes) we were able to identify factors that facilitated the project’s perceived personal and community benefits. We identified several factors that helped to facilitate positive outcomes. The partnership benefitted from an existing level of trust based on returning to work with the community following a previous project. Another facilitator was having a clearly defined health issue that was important to the community and created a common vision and focus for the project. The effective implementation of processes that established quality and equitable relationships among partnership members was a key factor.

Trust was a prominent theme in our analysis. Trust is known to strengthen partnership synergy and sustainability and is necessary for population-level outcomes such as systemic transformations, health improvement and spin-off projects (Jagosh et al. 2015; Lasker, Weiss & Miller 2001). Development of trust was facilitated by processes that promoted mutual respect, open dialogue and participatory decision making, as well as the long-term engagement and completion of the project objectives by the academic partners (Jagosh et al. 2015; Lucero 2013; Peralta, Smithwick & Torres 2020).

We used a mixed-methods approach to assess the outcomes of the project. These included gathering the perspectives of project participants through surveys and qualitative interviews based on the CBPR model, documentation of the outcomes of the work groups formed to implement prioritised strategies recommended by the community, and a Ripple Effects Mapping Exercise (REM) that allowed community members to reflect on the positive, negative and unintended impacts of the project. Despite the small number of participants, there was concurrence on the characteristics of the partnership as well as the tangible outcomes of the project between project participants that was evident in all three of the evaluation methods. Participants were excited about the tangible accomplishments of the work groups, and also enthusiastically described the importance of relationship building, trust and using a structured process. REM allowed for an exploration of unintended project outcomes. In particular, REM identified a reduction in stigma among providers and community residents that resulted in increased openness to others’ perspectives and a decrease in the isolation felt by many individuals and family members dealing with SUD.

The use of a structured method for engaging and directing community activities was credited with breaking down long-standing silos between community organizations and mobilizing the community for action. It was also credited with facilitating positive working relationships and building individuals’ skills and self-confidence. The SEED Method seeks to address key determinants identified as important to partnership synergy by Lasker, Weiss and Miller (2001) including the equitable use of resources, diverse stakeholder engagement in the partnership, cultivation of trust and mutual respect and a planned process for partnership administration leading to community action. Interview respondents indicated that the SEED Method provided a safe space where participants could get to know each other, overcome biases and learn to work together for the community good. Effective leaders were credited with facilitating this safe environment.

This study has limitations including the small sample size and the lack of participation in the interviews of community research team members who left before the end of the project. In addition, although the importance of relationship building, trust and a planned and structured method of engagement has broad application, evaluation results are specific to this project’s context which may limit generalizability.

Partnership structure impacts partner processes and synergy (Lasker, Weiss & Miller 2001; Weiss et al. 2002), which in turn impacts community-level outcomes (Khodjakov et al. 2011). Our evaluation confirmed satisfaction with partnership function and a high degree of partnership synergy, which undoubtedly contributed to the personal and community level outcomes. More recent research is attempting to determine the partnering practices within specific contexts that effectively contribute to research, community and health equity outcomes (Wallerstein at al. 2020). This case study provides a detailed description of the context of a community engaged project that followed a defined method grounded in the core principles of community engagement, as well as the tangible and perceived personal and community level outcomes identified through a mixed-methods evaluation approach. Although not evaluated in this study, we will incorporate measures of health equity outcomes in future projects. Our findings may inform the approach to engagement of other community-based participatory research projects, as well as their evaluation approach.