Co-creating a program theory and evaluability assessment for an Irish single-session, synchronous chat-based youth mental health intervention: implications for outcome evaluation

TL;DR

This paper explores how to evaluate a single-session online chat mental health service for youth in Ireland, focusing on program theory and contextual factors.

Contribution

It presents one of the first collaborative evaluability assessments for a chat-based mental health intervention.

Findings

Co-design workshops identified 12 core components influencing outcomes like accessibility and distress reduction.

A program theory model was developed to guide evaluation in complex, real-world contexts.

Contextual factors like collaboration and integration were found crucial for service success.

Abstract

Single-session online synchronous chat offers immediate, anonymous, single-session support for young people. However, the drop-in format attracts a diverse population with urgent and varied needs, creating challenges for evaluation. Standardized outcome measures may not capture short-term changes, and randomized controlled trials may be ethically inappropriate. These constraints point to the value of theory-based evaluation approaches rooted in implementation science, which can better capture short-term change, contextual complexity, and real-world variation in service delivery.

This study applied a theory-driven evaluability assessment to Jigsaw Live Chat, an Irish single-session, synchronous chat-based mental health service for youth. A situational analysis, review of the literature, and an internal data review established the intervention context and examined existing evaluative approaches within the literature. Two co-design workshops with staff (N = 13) identified contextual barriers, preconditions and assumptions for intervention effectiveness, core components, mechanisms of change, and intended outcomes. These elements were synthesized into a program theory, which informed the development of recommendations for evaluation.

Several contextual factors were identified as crucial to the success of Jigsaw Live Chat, including cross-collaboration, integration, and buy-in across youth mental health services. Staff defined a range of core components (N = 12) of the intervention, detailing how these components might influence various outcomes such as accessibility, ease of use, perceived usefulness, help-seeking intention, and decreases in immediate distress and overwhelm among young people. The broader impacts of the service were developed, reflecting the potential contribution of synchronous chat within a wider, integrated youth mental health care system nationally. These findings were integrated into a causal program theory model, and tailored indicators were proposed to support feasible and meaningful evaluation.

This study presents one of the first published examples of an iterative, collaborative evaluability assessment of a single-session synchronous chat-based mental health support service. The findings clarify how such interventions operate within specific organizational and sociocultural contexts and provide the foundation for future adaptation and evaluation aimed at improving service delivery and outcomes for young people.

Figures