The outpatient experience questionnaire for child and adolescent mental health services: reliability and validity following a nationwide survey

TL;DR

This study tested a questionnaire to measure patient experiences in adolescent mental health services, confirming its reliability and creating a shorter version for easier use.

Contribution

The study developed and validated a short version of the OPEQ-CAMHS questionnaire for adolescents aged 16 and above.

Findings

The OPEQ-CAMHS showed strong psychometric properties, including low missing data and valid factor structure.

A six-item short version was developed, maintaining reliability while reducing respondent burden.

The questionnaire demonstrated strong construct validity by correlating with self-reported well-being and mental state.

Abstract

The main objective was to evaluate the psychometric properties of the OutPatient Experience Questionnaire for Child and Adolescent Mental Health Services (OPEQ-CAMHS) among patients aged 16 and above, with a secondary objective of developing a parsimonious set of items and a short version of the instrument for this age group.

A national pilot study was conducted with adolescents from outpatient CAMHS in Norway, testing a new digital, continuous measurement approach using a measurement instrument developed through a comprehensive methodological framework. The study assessed missing data, ceiling effects, factor structure, internal consistency, discriminant validity, and construct validity. A shorter version was derived from psychometric results.

The pilot study included responses from 555 (46.3%) patients. Low proportions of missing or “not applicable” responses were found for 18 of the 20 items, and all items were below the ceiling-effect criterion. Two scales emerged: “structure and process”, and “outcome”, both meeting alpha criteria. Each individual item demonstrated a stronger correlation with its expected scale than with any of the other scales. Construct validity was confirmed through associations with relevant variables expected to be associated with patient-reported experiences, including self-reported current state and well-being. The results supported a six-item short version.

Psychometric testing confirmed data quality, internal consistency, and construct validity of OPEQ-CAMHS. The short version addresses respondent burden concerns and is now ready for broad implementation in Norwegian CAMHS and potentially in similar healthcare settings worldwide.

Figures

Item information functions for the OPEQ-CAMHS, Experiences with CAMHS - structure and processFig. 2Categorical response curves for the OPEQ-CAMHS, Experiences with CAMHS - structure and process

Full text

Background

Patient experiences are increasingly recognized as crucial for improving healthcare quality, alongside clinical effectiveness and patient safety [1–3]. A systematic review found consistent positive associations between patient experience, safety and effectiveness across various disease areas, settings, outcome measures and study designs [1].

Mental health disorders among adolescents pose economic and public health challenges with long-lasting consequences [4], with over one in ten experiencing mental illness at any given time [5]. However, there is limited knowledge about adolescent involvement in mental healthcare, and few studies have evaluated their experiences [4, 6, 7]. Fewer studies focus on younger populations, and gaps exist in understanding the use of patient-reported experience measures (PREMs) in paediatrics [8]. Little is also known about the routine use of PREMs to improve experience and outcomes in child and adolescent mental health services (CAMHS) [9].

Young people have unique health care needs and perceptions, necessitating specific measurement of their experiences [10]. Review findings highlight eight key domains crucial to young people’s care experiences, including clinician attitudes, communication, competency, involvement, guideline-driven care, outcomes, accessibility, and age-appropriate environments [11]. In mental health services, adolescents prioritize involvement in care decisions, privacy, age-appropriate information, and a trusting relationship with professionals [9]. Staff consistency is crucial in mental health services for adolescents, as it fosters trust and continuity. Frequent staff changes can disrupt the therapeutic relationship, making it difficult for adolescents to feel secure and supported [9].

The Norwegian Institute of Public Health (NIPH) conducts research on patient-reported experiences and outcomes, aiming for a more patient-centred health system through surveys and continuous measurements across various patient populations [12, 13]. Previously, NIPH has mainly focused on adult services, leaving a gap in understanding younger populations’ experiences. However, in recent years, instruments have been developed to measure young patients’ experiences with outpatient diabetes care and CAMHS [14, 15]. The NIPH has previously conducted national parent experience surveys with outpatient CAMHS in 2006 and 2017. Starting in 2023, NIPH is implementing digital, continuous electronic measurements, aligning with modernizations for other patient groups [12, 13]. Continuous measurements offer a novel, feasible and time-effective approach by collecting large-scale data via web-based administration methods, encompassing both parent/guardian and patient experiences in CAMHS [16]. A person-centred approach involving both children and parents yields valuable insights into healthcare quality [17].

This study aims to determine the data quality, validity and internal consistency reliability of the OutPatient Experience Questionnaire for Child and Adolescent Mental Health Services (OPEQ-CAMHS) among patients 16 years and older following a nationwide pilot. The instrument is designed to measure patient-experiences among adolescents aged 12–18 years visiting CAMHS in Norway. However, challenges related to obtaining parental consent and digital access issues for guardians of 12–15-year-olds limited participation in this age group, therefore, only those aged 16 and older are included in this study. Additionally, we aimed to develop a short version of the instrument focused specifically on PREMs. The OPEQ-CAMHS was tested applying standard NIPH methodology [12, 13, 18–20].

Methods

Measurement instruments

The development and validation of the OPEQ-CAMHS followed NIPH’s standard methodology, including a literature review, patient interviews, expert-group consultations, and pilot testing [15]. The instrument was originally developed to assess patient experiences during treatment in CAMHS. For this study, we adapted the instrument slightly by modifying verb forms to reflect a post-treatment context rather than during ongoing treatment. Additional File 1 presents the questionnaire.

The OPEQ-CAMHS comprises 40 closed-ended items, with 28 addressing patient experiences. Two open-ended questions inquire about experiences with medications while under the care of CAMHS, and overall experiences with CAMHS. The questionnaire is divided into sections covering therapists (13 items), CAMHS cooperation with others (5 items), medication (11 items), other questions about CAMHS including the effectiveness of support in managing daily life, satisfaction with treatment, feelings of coercion regarding CAMHS attendance, and the necessity of contacting CAMHS when starting treatment (4 items), and background questions (6 items). The final question in the survey asks who completed the survey: the respondent alone, together with parents/guardians, or parents/guardians by themselves.

Most experience items use a five-point response format ranging from 1 (not at all) to 5 (to a very large extent), with an additional option for “not applicable/don’t know”. This response scale is consistently applied in NIPH surveys, facilitating comparisons over time and among different healthcare user groups [12, 18–20]. Smiley face icons illustrate most response options.

The Five-item World Health Organization Well-being Index (WHO-5) is a widely used patient-reported outcome measure (PROM) designed to assess subjective well-being [21, 22]. It has been validated for screening depression and as an outcome measure in clinical trials, demonstrating applicability across various study fields [21]. In this study, the WHO-5 was selected to measure patients’ emotional well-being, comprising five positively worded items covering cheerfulness, calmness, activity, rest, and interest [23]. Respondents rate how well each statement applies to them over the last 14 days, with scores ranging from 5 (all of the time) to 0 (at no time), resulting in a raw score ranging from 0 (absence of well-being) to 25 (maximal well-being).

Additionally, a separate module addressing medication experiences within CAMHS was included in the instrument [24], but it is not part of the generic measure.

Data collection

The OPEQ-CAMHS was tested in a national pilot including patients at outpatient CAMHS in Norway, serving as the initial step to develop and validate new continuous measurements.

Data collection was conducted electronically between January-February 2023 and October-November 2023. Samples were obtained from the Norwegian Patient Registry (NPR), and participants were invited through Helsenorge.no, covering over 90% of Norway’s population. Inclusion criteria required patients to have ended CAMHS treatment in November 2022, August 2023, or September 2023. For children under 12, only parents were invited to complete the Parent experiences questionnaire for outpatient child and adolescent mental health services (PEQ-CAMHS Outpatients) [25].

The pilot included patients aged 12 and older. Patients under 16 required parental consent, while those 16 and older could provide their own consent. Due to access issues on the Helsenorge.no platform for guardians of 12–15-year-olds, it was necessary to invite guardians (and their children) via postal mail. As parental consent was required before inviting children aged 12–15, this limited the number of invited participants in this age group. Only 49 out of 595 eligible participants in this cohort were invited, and of these, 20 responded. Given the small number of respondents and the challenges associated with data collection in this age group, we chose to exclude the 12–15-year-olds from the current study. Therefore, this study focuses on patients 16 and older. Of 1,593 eligible patients in this age group, 1,200 were reached via Helsenorge.no (75.3%). Non-respondents received up to three SMS reminders. Due to a low response rate in the November 2022 group, a lottery incentive was introduced for the August and September 2023 groups (10 gift cards of NOK 5,000 each).

Statistical analysis

Missing data and ceiling effects were assessed, with items exceeding > 20% of missing data and “not applicable” responses excluded from factor analysis to minimize response loss [12, 19, 20]. Ceiling effects, indicating the percentage of respondents selecting the most favourable response option, were deemed acceptable if below 50% [12, 19, 20, 26].

Exploratory factor analysis (EFA) and Item Response Theory (IRT) were applied to evaluate instrument properties. Items with a five-point scale on outpatient CAMHS experiences were analysed. EFAs were conducted using principal-axis factoring to assess the underlying factor structure of the OPEQ-CAMHS, with promax rotation selected for interpretability. Factors with eigenvalues > 1 were rotated. Items with loadings < 0.40 or cross-loadings > 0.30 were excluded. The analyses combined data-driven and theoretical approaches, separating outcome items from process and structure items, consistent with previous psychometric testing in the national survey program. Following the guidance of Kilbourne et al., mental healthcare quality measures should be validated across Donabedian’s framework, encompassing structure, process, and outcomes [27]. This conceptual foundation guided the two-step factor analysis, as we aimed to measure different components of patient experiences in alignment with the program’s design, i.e., separating outcome items from process and structure items. Scale reliability was confirmed using item-total correlation and Cronbach’s alpha, with a minimum acceptable value of 0.7 [28]. To explore the potential influence of the lottery incentive introduced in the August/September sample, we conducted separate EFAs and assessed scale reliability for the November and August/September samples. The same procedures were applied to both groups as for the full sample.

To assess discriminant validity, we examined the correlations between each item and the scale it was hypothesized to belong to, as well as its correlations with other scales. The scales were scored by first recoding the individual item responses from a 1–5 scale to a 0–100 scale. We then computed an overall score for each scale by averaging the recoded items, with the requirement that at least half of the items must have valid responses for the score to be calculated. We expected the items to show stronger correlations with their respective hypothesized scales than with other scales. Spearman’s rho was used to compute these correlations.

Construct validity evaluates how well scores from health-related patient-reported outcomes (HR-PRO) instruments align with theoretical expectations, including internal correlations and group differences, ensuring accurate measurement of the intended construct [29]. In CAMHS, factors such as involvement in treatment decisions, sufficient time for questions, and satisfaction with appointment frequency are crucial for adolescent satisfaction [30], outweighing socio-demographic and disorder severity factors. For adult psychiatric inpatients, coerced admission and treatment, along with self-perceived current state, significantly influence outcome assessments [31]. This study hypothesizes correlations between background variables, including perceived need for CAMHS contact, coercion perception regarding CAMHS initiation, overall well-being assessment, and WHO-5 scores and scale scores, analysed using Spearman’s rank correlations. We expected small to moderate positive correlations between perceived need for CAMHS contact and patient experiences, as well as between patient experiences and WHO-5 scores. Conversely, we anticipated a small to moderate negative correlation between feelings of coercion and patient experiences.

IRT analyses enhanced our understanding of item performance within the OPEQ-CAMHS, crucial for developing its short version [32]. We applied the Generalized Partial Credit Model (GPCM) for its adaptability in estimating item-discrimination and category-response parameters, focusing on “structure and process” items. The scale contained sufficient items for IRT analyses, enabling detailed insights into each item’s function across the latent construct. Key analytical aspects included item discrimination (‘a’), item difficulty or location (‘b’), and the S − χ² item-fit statistic. A higher ‘a’ value indicates greater sensitivity to changes in the latent trait, while a lower value suggests less sensitivity. The S–χ² value assesses model fit for each item, with lower values indicating better fit and reliable measurement of the intended construct [33]. However, these fit indices are sensitive to sample size, with large samples potentially yielding poor fit even for trivial discrepancies [32]. Item difficulty parameters (b1 to b4) provide critical insights into the latent trait threshold at which respondents are likely to agree with each item. Higher negative values indicate easier endorsement, while lower values indicate greater difficulty.

Our main objective was to condense the OPEQ-CAMHS to its most impactful items for a concise version, considering missing data, ceiling effects, and insights from EFA and IRT. The consistency between the shortened and full version of the “structure and process” scale was assessed using intraclass correlation coefficients (ICCs). Statistical analyses were conducted using SPSS (version 28.0.1.0) and R (version 4.0.2), with specialized analyses performed using the lavaan, semPlot, and mirt packages.

The WHO-5 measure well-being using five items scored on a five-point scale, reflecting subjective well-being over a 2-week period [21, 22]. As scales measuring health related quality of life are conventionally translated to a percentage scale from 0 (absent) to 100 (maximal), it is recommended to multiply the raw score by 4 [21]. An index was computed by summing the evaluations, following previous research and using consistent measures [34].

Results

We received 133 responses (35.8%) from patients in November 2022. Response rates for the August and September 2023 groups were 52.1% and 49.9%, respectively. The NIPH received 555 completed questionnaires, achieving a total response rate of 46.3%. The vast majority (97.6%) were completed by the patients themselves, while a small proportion were completed either jointly with parents/guardians (2.2%) or by parents/guardians alone (0.2%). Girls comprised the majority of participants, accounting for 81.8% of the sample (n = 454) (Table 1). In terms of the age distribution, 16-year-olds made up 30.1% (n = 167), while 17-year-olds constituted 43.2% (n = 240) of the sample. Regarding perceived need for help from CAMHS, 44.4% (n = 232) felt they needed assistance “to a very large extent”, while 38.6% (n = 208) did not feel forced at all to seek help. When asked about their current well-being, 26.3% (n = 145) reported feeling “very poor” or “rather poor”. The mean WHO-5 score was 42.0 (SD = 22.8).Table 1. Background variables (n = 555)n%Sex Male10118.2 Female45481.8Age, years 1616730.1 1724043.2 1812622.7 19193.4 2030.5Did you feel that you needed contact with CAMHS? Not at all305.7 To a small extent346.5 To a moderate extent8616.4 To a large extent14127.0 To a very large extent23244.4Did you feel forced by others to start going to CAMHS? Not at all20838.6 To a small extent12122.4 To a moderate extent10319.1 To a large extent448.2 To a very large extent6311.7Overall, how are you feeling today? Very poor437.8 Rather poor10218.5 Both-and22340.5 Rather good12723.0 Very good5610.2WHO-555042.0/22.8 (mean/SD)

Table 2 presents the levels of missing data, responses in the “not applicable” option, mean values and ceiling effects for the 20 items related to patient experiences considered relevant for psychometric testing. Missing data ranged from 0.4% to 1.4%, and responses in the “not applicable” category varied from 0.0–38.2%. Except for two items, all OPEQ-CAMHS items had fewer than 20% missing or “not applicable” responses. The exceptions were item 16 (“Was the cooperation between CAMHS and other family members or your social circle good for you?”) (39.6%) and item 14 (“Were you able to talk to your therapist about things that were important to you when your parents/guardians were with you during your appointments?”) (20.2%). The mean score on the scale from 1 to 5, where 5 represented the best possible experience, was highest for item 1 (“Did you feel welcomed by your therapist?”) (4.08), and lowest for item 11 (“Did you and your therapist get to know each other well enough before talking about difficult subjects?”) (2.90). All items met the ceiling-effect criterion (<50% in the most-positive response option), and none were excluded from further analyses due to this criterion.Table 2. Item descriptions for the OPEQ-CAMHSnMissing (%)Not applicable (%)MeanCeiling (%)Patient experiences1Did you feel welcomed by your therapist?5520.5-4.0838.42Did you get enough time with your therapist?5481.3-3.4217.23Did your therapist provide satisfactory answers if you had any questions?5310.44.03.5021.54Did your therapist have good suggestions for how you could both work together?5330.53.43.4720.65Did your therapist seem to be good at their job?**5440.71.33.8034.66Did you feel that your therapist listened to what you had to say?**5510.7-3.7634.87Did your therapist seem to understand how you were feeling?5410.71.83.2918.18Did you feel that your therapist cared about you?**5410.52.03.7532.09Did you feel safe with your therapist?5450.51.33.6128.410Were you able to talk to your therapist about things that were important to you?**5410.71.83.6027.411Did you and your therapist get to know each other well enough before talking about difficult subjects?5290.74.02.909.812Were you able to take part in making decisions about your treatment?5150.96.33.2920.213Were you able to take part in making decisions about what happened during your appointments?5180.75.93.2916.614Were you able to talk to your therapist about things that were important to you when your parents/guardians were with you during your appointments?5431.318.93.1416.915Was the cooperation between CAMHS and your parents/guardians good for you?5620.716.03.3218.816Was the cooperation between CAMHS and other family members or your social circle good for you?3351.438.23.1616.717Was the cooperation between CAMHS and your school PPT (psychological counselling service), your GP, or other services good for you?4730.913.93.2216.718Did you feel that you were given the opportunity to decide what your therapist shared about you with others CAMHS was in contact with?4820.912.33.6530.130Overall, did you get the help you needed to better manage everyday life?**5310.73.62.9815.431Overall, did you get good help and treatment from CAMHS?**5491.10.03.1416.3Most items were scored on a 5-point response scale ranging from 1 (“not at all”) to 5 (“to a very large extent”)**Items finally selected for the short version of the instrument

The first EFA included the 16 items related to “structure and process”. The two “outcome” items were included in the second analysis. No items were excluded due to low factor loading (<0.40), or cross-loadings > 0.30. The EFA for “structure and process” items identified one factor explaining 63.3% of the variance (Table 3). The second EFA included the “outcome” items and identified one factor explaining 92.9% of the variance. Item-total correlations for both scales were acceptable (0.65 to 0.86), and Cronbach’s alpha values ranged from 0.92 to 0.96, meeting the criterion (0.70) (Table 3). To examine the potential impact of the lottery incentive on factor structure and scale reliability, we conducted separate analyses for the November and August/September groups. The results indicated consistent factor structures across both samples, with one factor identified for “structure and process” items and one for “outcome” items in each group (results not shown). Cronbach’s alpha values were acceptable across both samples, and the variance explained by the factors remained stable.Table 3. Factor loadings and reliability statistics for the OPEQ-CAMHSFactor loading*Corrected item-total correlationCronbach’s alphaCronbach’s alpha if item deletedExperiences with CAMHS - structure and process0.9610Were you able to talk to your therapist about things that were important to you?0.860.850.968Did you feel that your therapist cared about you?0.860.840.966Did you feel that your therapist listened to what you had to say?0.860.840.969Did you feel safe with your therapist?0.850.830.964Did your therapist have good suggestions for how you could both work together?0.850.830.965Did your therapist seem to be good at their job?0.850.820.967Did your therapist seem to understand how you were feeling?0.840.820.963Did your therapist provide satisfactory answers if you had any questions?0.840.810.961Did you feel welcomed by your therapist?0.810.790.9611Did you and your therapist get to know each other well enough before talking about difficult subjects?0.770.750.9613Were you able to take part in making decisions about what happened during your appointments?0.700.690.9612Were you able to take part in making decisions about your treatment?0.670.660.9615Was the cooperation between CAMHS and your parents/guardians good for you?0.670.660.962Did you get enough time with your therapist?0.670.660.9618Did you feel that you were given the opportunity to decide what your therapist shared about you with others CAMHS was in contact with?0.670.660.9617Was the cooperation between CAMHS and your school PPT (psychological counselling service), your GP, or other services good for you?0.660.650.96Experiences with CAMHS - outcome0.9230Overall, did you get the help you needed to better manage everyday life?0.860.86-31Overall, did you get good help and treatment from CAMHS?0.860.86-*Separate factor analysis for i) structure ad process items, and ii) outcome items

Table 4 shows that all items exhibited stronger correlations with their respective scale than with the other scale, with the item-to-own-scale correlation coefficients ranging from 0.66 to 0.97. All correlations were statistically significant (p < 0.001).Table 4. Correlations between items and scales for the OPEQ-CAMHSStructure and processOutcomeExperiences with CAMHS - structure and process1Did you feel welcomed by your therapist?0.810.652Did you get enough time with your therapist?0.660.573Did your therapist provide satisfactory answers if you had any questions?0.850.734Did your therapist have good suggestions for how you could both work together?0.850.725Did your therapist seem to be good at their job?0.850.716Did you feel that your therapist listened to what you had to say?0.840.697Did your therapist seem to understand how you were feeling?0.840.738Did you feel that your therapist cared about you?0.840.689Did you feel safe with your therapist?0.830.6810Were you able to talk to your therapist about things that were important to you?0.860.7011Did you and your therapist get to know each other well enough before talking about difficult subjects?0.790.6212Were you able to take part in making decisions about your treatment?0.740.6213Were you able to take part in making decisions about what happened during your appointments?0.750.5815Was the cooperation between CAMHS and your parents/guardians good for you?0.700.6317Was the cooperation between CAMHS and your school PPT (psychological counselling service), your GP, or other services good for you?0.660.5718Did you feel that you were given the opportunity to decide what your therapist shared about you with others CAMHS was in contact with?0.700.50Experiences with CAMHS - outcome30Overall, did you get the help you needed to better manage everyday life?0.780.9731Overall, did you get good help and treatment from CAMHS?0.820.96Correlations in bold show item to own scale correlations

Table 5 shows that all tests of construct validity were statistically significant. The correlation between patients’ perceived need for CAMHS contact and their experiences with CAMHS was significant, with stronger perceived need correlating with more positive experiences. Scale scores correlated weakly to moderately with feelings of being coerced into CAMHS and self-reported general condition, ranging from − 0.18 to 0.41. Lower coercion levels and better general condition were associated with higher scale scores. Higher WHO-5 scores, which reflect better psychological well-being, were significantly correlated with more positive patient experiences.Table 5. Construct validity testing: associations between scales, background variables and responses to individual questionnaire itemsStructure and processOutcomeDid you feel that you needed contact with CAMHS?0.230.27Did you feel forced by others to start going to CAMHS?−0.23***−0.18Overall, how are you feeling today?0.330.41WHO-50.330.40******p < 0.001; **p < 0.01; *p < 0.05; ns, not significant. Data are represented as Spearman’s rank correlation coefficients

In the IRT analysis of the OPEQ-CAMHS “structure and process” scale, we examined item performance, discrimination, difficulty levels, and sensitivity to patient experiences. Key findings highlight variations in item fit, discrimination parameters, and difficulty thresholds, as well as identification of items that are highly sensitive to differences in patient experiences. It is important to note that this analysis is applied to the “structure and process” scale due to the limited number of items in the “outcome” scale.

Table 6 displays the IRT analysis parameters including discrimination (a), difficulty thresholds (b1–b4), and item fit (S-χ2 and p-values) for the OPEQ-CAMHS “structure and process” scale. The S − χ^2^ varied from 32.21 to 93.79, with all items showing satisfactory performance based on p-values. Item 15 (“Was the cooperation between CAMHS and your parents/guardians good for you?”) exhibited the highest S–χ² value at 93.79, but the corresponding p-value suggest an adequate fit. Item 3 (“Did your therapist provide satisfactory answers if you had any questions?”) had the lowest S–χ² value of 32.21, indicating a good fit to the model.Table 6. Parameter estimates derived from IRT analysis of the OPEQ-CAMHS, Experiences with CAMHS - structure and processab1b2b3b4S– \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\chi }}^2}$$\end{document} pExperiences with CAMHS - structure and process1Did you feel welcomed by your therapist?2.87−1.97−1.60−0.880.2637.920.3822Did you get enough time with your therapist?1.10−1.94−1.10−0.251.1660.690.5943Did your therapist provide satisfactory answers if you had any questions?2.63−1.72−1.13−0.120.8132.210.8864Did your therapist have good suggestions for how you could both work together?2.76−1.72−0.99−0.160.8461.150.0555Did your therapist seem to be good at their job?3.03−1.48−1.27−0.510.2857.580.0996Did you feel that your therapist listened to what you had to say?2.83−1.38−1.16−0.460.2947.170.3847Did your therapist seem to understand how you were feeling?2.37−1.18−0.92−0.010.8847.430.6168Did you feel that your therapist cared about you?3.21−1.57−1.12−0.470.3849.760.2559Did you feel safe with your therapist?2.77−1.43−1.14−0.380.5348.050.27510Were you able to talk to your therapist about things that were important to you?2.97−1.64−1.06−0.230.5338.800.52411Did you and your therapist get to know each other well enough before talking about difficult subjects?1.43−0.85−0.570.391.4363.010.61612Were you able to take part in making decisions about your treatment?0.92−1.21−0.91−0.050.9876.180.68913Were you able to take part in making decisions about what happened during your appointments?1.15−1.72−0.830.071.1883.540.09715Was the cooperation between CAMHS and your parents/guardians good for you?1.00−1.28−1.270.131.0193.790.10717Was the cooperation between CAMHS and your school PPT (psychological counselling service), your GP, or other services good for you?0.93−1.39−1.070.301.0366.950.88618Did you feel that you were given the opportunity to decide what your therapist shared about you with others CAMHS was in contact with?1.00−1.17−1.17−0.770.6685.240.4120.92 to 3.21–1.97 to –0.85–1.60 to –0.57–0.88 to 0.390.26 to 1.43a: discrimination; b1–b4: thresholds. S– \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\chi }}^2}$$\end{document} represents item fit statistics, with P values < 0.05 indicating lack of fit

The ‘a’ values, reflecting an item’s discrimination ability, ranged from 0.92 to 3.21. Item 8 (“Did you feel that your therapist cared about you?”) displayed the highest ‘a’ value of 3.21, indicating high sensitivity to patient experience variation, closely followed by item 5 (“Did your therapist seem to be good at their job?”) with 3.03. Conversely, items 12 (“Were you able to take part in making decisions about your treatment?”) and 17 (“Was the cooperation between CAMHS and your school PPT (psychological counselling service), your GP, or other services good for you?”), had the lowest ‘a’ values (0.92 and 0.93 respectively), indicating lower discrimination effectiveness.

The difficulty parameters, represented by b1 to b4, reveal the threshold at which respondents are inclined to agree with various questionnaire items (Table 6). The b1 range from − 1.97 to − 0.85 indicates varied difficulty levels across items. Similarly, for b2, b3, and b4, the ranges were − 1.60 to − 0.57, − 0.88 to 0.39, and 0.26 to 1.43, respectively. The variation in these difficulty parameters suggests that the questionnaire captures a diverse range of experiences and could be sensitive to varying levels of patient experiences.

Figures 1 and 2 illustrate the curves and response patterns. Items 1, 5, 6 (“Did you feel that your therapist listened to what you had to say?”) 8, 9 (“Did you feel safe with your therapist?”) and 10 (“Were you able to talk to your therapist about things that were important to you?”) demonstrates steep curves, indicating heightened sensitivity to changes in the latent trait level. Conversely, items 2 (“Did you get enough time with your therapist?”), 12, 13 (“Were you able to take part in making decisions about what happened during your appointments?”), 15 and 17 display flatter curves, suggesting they are less sensitive to changes in the trait level, providing consistent information across a wider range of the trait (Fig. 1). Additionally, the categorical response curves (CRCs) in Fig. 2 highlight questionable values for the second response category, particularly for items 15 and 18 (“Did you feel that you were given the opportunity to decide what your therapist shared about you with others CAMHS was in contact with?”).Fig. 1. Item information functions for the OPEQ-CAMHS, Experiences with CAMHS - structure and processFig. 2Categorical response curves for the OPEQ-CAMHS, Experiences with CAMHS - structure and process

Four items were selected for the abbreviated version after comprehensive evaluation of their psychometric properties.

The EFA of the “structure and process” scale revealed high factor loadings for several items, notably items 10, 8, and 6 (Table 3). Additionally, IRT analysis informed our selection, with items 3, 1, and 10 showing the best fit according to the S − χ2 statistic, and items 8, 5, and 10 demonstrating the strongest discriminative ability (Table 6). Items 1, 5, 6, 8, 9, and 10 outperformed others in terms of ideal CRCs.

Items 5, 6, 8, and 10 were chosen, covering key themes such as therapist competence, patient’s sense of being heard and cared for, and the opportunity to discuss significant matters. The ICC comparing the full 16-item scale with the 4-item short form, was high at 0.97 (p < 0.001), indicating strong consistency.

To sum up, item reduction was conducted based on psychometric properties evaluated through EFA and IRT analysis. The process included assessing factor loadings, cross-loadings, and item-total correlations. No items were excluded solely based on low factor loadings. After careful evaluation of all properties and the item fit statistics derived from the IRT model, four items of the “structure and process” scale along with both “outcome” scale items were ultimately selected for the abbreviated version of the instrument.

Discussion

This study aimed to evaluate the psychometric properties of the OPEQ-CAMHS through a nationwide pilot of continuous electronic measurements in Norway during 2022 and 2023. Additionally, it sought to provide insights into item performance and develop a shorter version of the instrument.

The psychometric testing confirmed good data quality and internal consistency. The OPEQ-CAMHS consists of two scales based on both empirical and theoretical assumptions, covering evaluations of “structure and process”, and “outcome”. The content of the scales aligns with key aspects identified in reviews of PREMs in mental health care, highlighting interpersonal, trusting relationships, patient involvement, psychological care, access, care coordination, information, and outcomes [4, 9, 11, 35–38].

The results revealed a stable and interpretable scale structure, with low rates of omitted answers and few responses in the “not applicable” option, indicating the relevance of the questions to most patients. Patient involvement in the instrument’s development, including identifying critical aspects of outpatient care, ensured that it addresses a broad range of specific domains relevant for service improvement [15].

The findings demonstrate that each item is closely associated with its intended construct, reflecting accurate measurement. Patient experiences correlate with factors such as perceived need for CAMHS contact, coercion perceptions, current condition, and psychological well-being measured by the WHO-5. Higher satisfaction levels are associated with lower coercion and better health, emphasizing the importance of addressing individual patient needs, involving patients in treatment decisions, and attending to overall psychological well-being to enhance care quality. Prior research and theoretical frameworks offered limited hypotheses, but these findings align with earlier studies, suggesting that patient experiences in CAMHS are significantly influenced by subjective perceptions and psychosocial contexts rather than solely clinical or demographic factors [30, 31, 39]. Given the limited number of studies, a comprehensive examination of factors such as treatment modalities, personal perceptions, communication preferences, and the effects of interventions is necessary to better understand patient experiences in CAMHS. Despite increased PROMs use in child mental health, there remain inconsistencies in the measures used, highlighting the need for consensus on appropriate measures [40]. This reflects a broader challenge in the field, as there is currently no consensus on the most appropriate or standard PROMs for this population. Higher WHO-5 scores correlate with improved patient experiences, supporting its criterion validity. Further evaluation, including changes from baseline scores, is warranted to assess the relevance of subjective well-being in evaluating care quality in this setting.

Moreover, the application of IRT enriched our understanding of item performance, identifying variations in discrimination and difficulty among items. This guided selection for a shorter version of the instrument, maintaining strong psychometric properties. The national reference group for the project (representatives from patient organisations and clinicians from CAMHS) emphasized the need for shorter versions for children and adolescents. The objective of refining the scale was to develop a short version capable of effectively capturing variations and temporal changes among different groups. We carefully evaluated items from the “structure and process” scale, considering psychometric properties, missing response rates, and ceiling effects. Four items were chosen for inclusion in the abbreviated version, along with both “outcome” scale items. The abbreviated OPEQ-CAMHS comprises six items, addressing perceived outcomes, therapist competence, feeling listened to, discussing important matters, and feeling cared for. Our findings show that the instrument can be effectively reduced from its original 18-item format to a concise 6-item version, crucial for minimizing respondent burden while maintaining precision. Depending on the purpose of the study and the setting in which the measurement is conducted, there may be varying needs for including background variables or medication-related questions in addition to the 6-item PREMs version.

Furthermore, IRT results showed that the response scale for some items probably could be reduced to 3- or 4-levels. However, the 5-point scale is standard for all patient groups, so changing it for some of the questions would deteriorate the possibility for cross-item/scale comparisons and cross-patient comparisons in quality evaluation and improvement work. Also, the cognitive interviews in the development project showed that the five-point Likert scale functioned well, with most participants finding it easy to use and suitable for their needs. Having several different response scales in the questionnaire could also be more cognitively challenging for respondents. However, as the standard number of response categories for adults not necessarily suits younger patients, which was indicated by the IRT findings in the present study, we point to the need for more research on response formats for adolescents.

Notably, combining of EFA and IRT yielded valuable and distinct insights into OPEQ-CAMHS performance. The benefits of the shortened questionnaire include potentially improved response rates and reduced respondent burden, and its brevity allows adaptability for diverse populations and compatible with other assessments, enhancing versatility in various research and healthcare settings.

The nationwide pilot of continuous digital measurement supports the future integration of patient-reported experiences into national healthcare quality monitoring and improvement efforts. It provides a unique opportunity for standardized and research-based measurement of all patients discharged from CAMHS in Norway. The high response rate among this young patient group is promising, given the challenges of reaching and motivating them to respond to mental health surveys [41]. While around 36% responded in the November 2022 group, around 50% responded in the other groups after including a lottery incentive. A high response rate secure feedback from a substantial proportion of all patients using CAMHS about their health care experiences, self-reported mental health and well-being. Positive experiences correlated with better adherence to preventive and treatment processes, improved clinical outcomes, enhanced patient safety, and reduced healthcare utilization [2]. This underlines the importance of measuring, monitoring and improving patients’ experiences. The OPEQ-CAMHS is now ready for large-scale implementation in Norwegian CAMHS, with potential applicability to similar healthcare systems in other countries, albeit with adjustments to technical and legal considerations.

Published evidence on adolescents’ participation in surveys about their mental healthcare experiences and outcomes is limited. Future research should aim to improve both the quality and quantity of survey data to better understand and assess adolescents’ healthcare experiences and outcomes. Our study emphasizes the potential and importance of involving adolescents in providing feedback on healthcare issues, identifying problem areas, and suggesting improvement priorities. Adolescents value being included in discussions about their mental health treatment and decision-making processes [9].

The OPEQ-CAMHS provides actionable feedback for outpatient departments, aiding in performance monitoring and identifying areas for patient-centred quality improvement. While adolescent feedback is crucial, input from parents, who are integral to treatment, is also valuable. However, for legal reasons, patients 16 years or older must first consent to involving parents. This is an avenue for further development and research.

Strengths and limitations

This study focused on determining the data quality, validity and internal consistency reliability of the newly developed OPEQ-CAMHS questionnaire, demonstrating its reliability and feasibility for continuous electronic surveys in CAMHS.

Web-based surveys offer cost-effective advantages. Furthermore, to enhance the study’s credibility, the survey was administered by an impartial third-party entity, the NIPH, unaffiliated with healthcare provision. Exclusions based on non-registration at Helsenorge.no pose a limitation, warranting initiatives for improved coverage and research on differences between covered and non-covered patients, and follow-up studies of non-covered patients with other modes (postal, onsite, telephone etc.).

While the abbreviated questionnaire may lack comprehensiveness, it suits scenarios prioritizing respondent burden minimization. Depending on the specific context and objectives, researchers and healthcare practitioners may find it necessary to employ either the complete questionnaire or selectively incorporate items as needed [42].

This study investigates the experiences of patients within the cohort aged 16 and older. However, the OPEQ-CAMHS instrument was also developed with younger patients in mind. It has been tested for individuals aged 12–18 through qualitative interviews to identify key aspects of care, cognitive interviews, and a pilot study [15]. Future research should include psychometric evaluations for patients aged 12–15, potentially adapting the scale with additional items for younger children, thereby expanding the instrument’s applicability across a broader age range. To reach patients 12–15 years, a tailored strategy is required, where parental consent is necessary.

Conclusions

In conclusion, the OPEQ-CAMHS demonstrates satisfactory internal consistency, reliability, and validity, making it suitable for large-scale implementation in Norwegian CAMHS. The abbreviated version is valuable for reducing respondent workload. When engaged in continuous measurements, it’s crucial to recognize the validation process as an ongoing endeavour rather than a one-time procedure.

Psychometrically, the OPEQ-CAMHS shows robust characteristics, with EFA supporting a unidimensional structure for both the structure-process and outcome items as well as satisfactory internal consistency and construct validity.

This study contributes valuable insights to the literature on patient experiences in CAMHS, providing a rigorous appraisal of a tool for assessment and valuable support to both healthcare practitioners and researchers in their pursuit of patient-centred care within the realm of child and adolescent psychiatry. The questionnaire and data collection approach are adaptable for similar healthcare systems in other countries.

Further studies may benefit from diversifying the sample and incorporating longitudinal data, facilitating a deeper exploration of the nuanced experiences and perceptions of young individuals’ experiences in mental health care settings.