User Satisfaction with Child and Adolescent Mental Health Services: Factor Structure of the Experience of Service Questionnaire (ESQ) in Norway and the UK

TL;DR

This study examines how users in the UK and Norway rate their satisfaction with mental health services for children and adolescents.

Contribution

The study provides new evidence supporting a general satisfaction factor in the ESQ questionnaire for CAMHS users.

Findings

The ESQ showed strong psychometric properties in all samples.

A bifactor model with general and specific satisfaction factors best fit the data in most cases.

The Norwegian adolescent version was best represented by a unidimensional model.

Abstract

Child and Adolescent Mental Health Services (CAMHS) are expected to track user satisfaction routinely, and to this end, the Experience of Service Questionnaire (ESQ) is increasingly being adopted worldwide. The literature is inconsistent concerning the underlying factor structure of satisfaction measures, and debate is ongoing regarding the evidence of a general satisfaction factor.

This study aimed to examine the factor structure and dimensionality of the parent/carer and adolescent versions of the ESQ in the UK and Norway.

Data were retrieved from routine CAMHS clinical practice in the UK and Norway. Three models suggested by the research group were tested through Confirmatory Factor Analysis (CFA) and reliability testing.

A series of CFAs revealed sound psychometric properties of the ESQ in all samples. A bifactor model with a general satisfaction factor and two specific factors of Satisfaction with Care and Satisfaction with Environment fitted the data best, except for the Norwegian adolescent version where a unidimensional model was kept.

The results support the continued use of the ESQ in CAMHS in the UK and Norway and significantly contribute to the literature on user satisfaction by adding evidence of a general satisfaction factor.

Figures

Bifactor model of the ESQ. SWC = Satisfaction with care; SWE = Satisfaction with Environment

Full text

Introduction

Assessing user satisfaction is increasingly important in evaluating child and adolescent mental health services (CAMHS) worldwide (Ayton et al., 2007; Bear et al., 2022; Lebow, 1982). With the growing acceptance, initiatives regarding accurately measuring the construct of user satisfaction are being acknowledged (Biering, 2010). However, the construct validity of common user satisfaction measures in CAMHS across different countries remain underexplored.

Focusing on understanding families’ experiences and level of satisfaction not only enhances sustained engagement and reduces premature dropouts but is also recognized as a key factor in achieving optimal clinical outcomes (Rickwood et al., 2017). The impact of prioritizing this understanding is evident in clinical practices, at least in the UK. A decade ago, merely 5% of services systematically collected user satisfaction data (Batty et al., 2013; Hall et al., 2014). Recent research, however, indicates a notable shift, with 68–81% of practitioners now routinely incorporating user satisfaction measures in care (Bear et al., 2022). Compared to adult mental health where at least thirty measures of user satisfaction are available (Miglietta et al., 2018), a limited number of user satisfaction measures apply to CAMHS, including tools by Stüntzner-Gibson et al. (1995), Brannan et al. (1996), Garland, Saltzman, et al. (2000), Attride-Stirling (2002), Ayton et al. (2007), Day et al. (2011), and Haugum et al. (2019). Nevertheless, this growing literature emphasizes the value of families’ opinions regarding their satisfaction with services (Ayton et al., 2007; Barber et al., 2006; Biering & Jensen, 2010; Davison et al., 2017; Day et al., 2011; Rickwood et al., 2017; Solberg et al., 2015). Simultaneously, the literature also embodies perspectives representing a common criticism regarding user satisfaction measures often yielding a too general focus on satisfaction and possibly artificially excessive levels of satisfaction (Brannan et al., 1996; Crawford & Kessel, 1999; Kapp et al., 2017; Wolpert et al., 2016). It is important to consider patient-centred measures of user satisfaction that are customized to the unique needs of children, adolescents, and their parents/carers in order to address the complexity involved in CAMHS effectively (Brown et al., 2014). It has been suggested that the lack of measures with such qualities is a barrier to quality improvement in mental health services (Kilbourne et al., 2018).

The Experience of Service Questionnaire (ESQ, formerly known as “CHI-ESQ”) (Attride-Stirling, 2002) is an accessible user satisfaction measure increasingly being adopted in CAMHS in and outside of the UK (Arnesen et al., 2023; Bear et al., 2022; Brown et al., 2014; Bunge et al., 2014; Clark et al., 2018; Derby, 2016; Karagiorga et al., 2024; Khan et al., 2023; Kilburn et al., 2019; Lindevall, 2020; McGrath et al., 2024; Ozer & Halfon, 2024). Originally, this 12-item measure was developed for use across child health care in the UK to verify service delivery anonymously, but nowadays in line with evidence-based practice, the ESQ is recommended for use routinely with other core measures to ensure families’ experiences with the service will be monitored alongside any changes in symptoms or functioning (CORC, 2024b).

Quantitative measures, such as the ESQ, typically reveal overall high satisfaction with services among most families seen at CAMHS (Crawford & Kessel, 1999; Kapp et al., 2017; Wolpert et al., 2016). Patients often provide qualitative descriptions that highlight both the negative and positive aspects of their experiences. However, it’s important to note that these descriptions often contain valuable insights that can help improve patient care (Biering & Jensen, 2010; Crawford & Kessel, 1999). Findings have indicated that families who are satisfied with the service show higher treatment compliance, which in turn enhances both the clinical and social outcomes of care (Fitzpatrick & Hopkins, 1993; Mahin et al., 2004). By promoting satisfaction, services can hopefully reduce the risk of premature termination of treatment or disagreement between families and clinicians regarding care, improving mental health outcomes (Barber et al., 2006; Bjørngaard et al., 2008; Davison et al., 2017; Day et al., 2011; De Haan et al., 2013).

Implementing user satisfaction measures in routine clinical practice faces a persistent challenge: a lack of well-documented measures with adequate psychometric properties (American Educational Research Association, 2014). Many studies rely on reported internal consistency without examining the factor structure (Brown et al., 2014; Young et al., 1995). A review of the literature on quality in satisfaction measures in adult services concludes this problem still endures (Sanchez-Balcells et al., 2018). In a critical review looking into studies of adolescents with CAMHS experience, Biering (2010) delineated three universal factors of satisfaction: satisfaction with the service environment, clinician relationship, and treatment outcome. Biering (2010) underscored the importance of exploring the weak to moderate correlation between child and parent/carer satisfaction and urged researchers to consider previous research when studying satisfaction. Moreover, in the literature on user satisfaction in CAMHS, a pattern emerges where most studies focus on the development of new measures or adaptations of measures from adult mental health services. Acknowledging this, both Biering (2010) and Brown and colleagues (2014) note that research on the ESQ is one of the few satisfaction measures developed in conjunction with prior research on satisfaction.

In a large-scale study in the UK, the original ESQ demonstrated sound psychometric properties for the child, adolescent and parent/carer versions, and it was also found to be a reliable measure of satisfaction that distinguished between services (Brown et al., 2014). Others (Bunge et al., 2014; Davison et al., 2017; Ozer & Halfon, 2024) have corroborated the usefulness of the ESQ in various clinical settings. Originally, a sum-score determined the overall level of satisfaction (Attride-Stirling, 2002; Barber et al., 2006), leaving little room for understanding the drivers of differences in satisfaction. More recently, Brown and colleagues (2014) elaborated on this by revealing a two-factor structure with most items loading on a factor of Satisfaction with the Care provided and the remaining three items loading on Satisfaction with Environment.

Brown and colleagues (2014) identified strong evidence for a two-factor solution with items loading on the factors Satisfaction with Care and Satisfaction with the Environment. They also found common variance between these two factors, tested by exploratory factor analysis, and suggested a strong “halo” effect. This “halo” effect was considered indicative of responses to the ESQ, as with other satisfaction measures, underpinned by a general attribute of satisfaction. Namely that service users’ overall feelings, or general impression, of satisfaction or dissatisfaction characterize their responses to each item in the ESQ. Previous research (Ayton et al., 2007; Brannan et al., 1996; Brown et al., 2014; Garland, Saltzman et al., 2000) suggest evidence of a general satisfaction factor, as specific factors are typically strongly correlated. However, the question of uni- vs. multi-dimensionality has yet to be examined using a bifactor model.

Having access to well-established, standardized and feasible measures is key to comprehensive coverage both clinically and for research purposes in any cultural context (De Vries et al., 2018). As such, translations of existing measures are preferable to the development of language-specific measures (Hafkenscheid et al., 2010). Despite the ESQ being in use across the world, to the best of our knowledge, the only non-English versions of the ESQ psychometrically examined are the Spanish version (Bunge et al., 2014), the Turkish version (Ozer & Halfon, 2024), and the Greek version (Kotsis et al., 2024). The Spanish version of the ESQ was found to be viable for a population selected from private CAMHS in Buenos Aires, but parallel to findings from Sanchez-Balcells et al. (2018) here too results solely relied on the acceptability of reported Cronbach’s Alpha and did not examine for factor structure. The Turkish version affirmed the two-factor solution by Brown et al. (2014). To sustain the growing application of the ESQ in CAMHS worldwide it is crucial to assess whether it accurately captures the intended construct by examining the ESQ factor structure across samples and countries.

Aims and Objectives

Despite user satisfaction being referred to as a general construct, the only study to have examined the bifactor structure of the ESQ is Kotsis et al. (2024), who found support for a unidimensional factor structure in the Greek version. Given the limited research on the topic, this study aims to add to the understanding of user satisfaction as a construct by further examining the factor structure of the ESQ in Norwegian and UK clinical samples, exploring unidimensional, two-factor, and bifactor solutions. Subsequently, we estimate the reliability of the factor structure with the best fit. Our hypothesis, based on preliminary findings of a general factor for satisfaction as noted by Brown et al. (2014), and influenced by previous research (Attride-Stirling, 2002; Ayton et al., 2007; Barber et al., 2006; Biering, 2010Brown et al., 2014; Day et al., 2011; Garland et al., 2000a, b; Kotsis et al., 2024), is that the bifactor solution would provide the best model fit to the empirical data.

Methods

Dataset

The sample for the current paper included families receiving support from CAMHS clinics in Norway and the UK. All included clinics are members of the learning collaborative, the Child Outcomes Research Consortium (http://www.corc.uk.net/). In the included clinics, routine outcome and satisfaction measures are collected as part of larger audits or service evaluations, where both adolescents, parents/carers and clinicians are invited to respond. Both the Norwegian and UK clinics systematically collected the data at two distinct time points using consistent procedures.

Data from the Norwegian sample was collected between December 2013 to December 2016, from one outpatient clinic at CAMHS at the University Hospital of Northern Norway (UNN). ESQ responses from adolescents and parents/carers were digitally collected. A total of 1,205 eligible patients were included.

The UK sample draws on data collected between February 2011 and December 2021 from multiple CAMHS across the UK and has a mix of digital and paper-completed responses, which were submitted annually to CORC’s central research team. A total of 9,761 parents/carers and 10,207 children and young people were included.

Measures

User satisfaction was assessed both from the adolescent and parent/carer perspective with separate versions of the ESQ, the original ESQ in the UK and the Norwegian translated version in Norway. Currently, no confirmed information has been available regarding the translation procedure for the Norwegian version of the ESQ other than that it is likely that the ESQ was translated to Norwegian in conjunction with a longitudinal epidemiological study in Norway between 2001 and 2012 (Heiervang et al., 2007).

The Experience of Service Questionnaire (ESQ) (Attride-Stirling, 2002) is a freely available questionnaire for exploring user satisfaction. Developed through focus groups with children and parents/carers across the child health sector, the ESQ exists in versions tailored to children (9–11), adolescents (12–18) and for parents/carers. The ESQ comprises 12 items rated on a three-point Likert scale: 1= “Certainly true”, 2 = “Partly true”, and 3 = “Not true”, with lower scores indicating higher satisfaction and higher scores indicating greater dissatisfaction. This scoring method aligns with the CORC Snapshot approach (CORC, 2024a). While alternative scoring methods exist, including the CORC + format (1 = “Not true”, 2 = “Partly true”, and 3 = “Certainly true”), which is also required in the NHS Digital Mental Health Services Data Set (MHSDS), and the Item Response Theory (IRT) approach (which codes responses as 0 = “Not true,” 1 = “Partly true,” and 2 = “Certainly true”), we retained the original scoring to maintain consistency with the instruments design as well as the Norwegian dataset, which builds on the CORC Snapshot approach. The ESQ also includes a “Don´t know” response option, which was treated as a non-substantive response and considered missing data in the analysis. Additionally, there are three open-ended questions allowing for free text responses; however, these were not analysed in this study.

Brown and colleagues (2014) found evidence for two highly correlated, but separate, factors named Satisfaction with Care and Satisfaction with Environment, using a two-level latent trait model. The Care factor (Q1-7 and 11–12) has a range from 9 to 27, and the Environment factor (Q8-10) has a range from 3 to 9. Lower scores indicate a higher degree of satisfaction. In the clinic, families respond to the ESQ at evaluation or discharge. Brown et al. (2014) found the ESQ to both be a valid subjective measure of CAMHS experiences and to reliably distinguish between services (Garralda et al., 2000; Goodman, 2001; Gowers et al., 1999; Hanssen-Bauer et al., 2007a; Hanssen-Bauer et al., 2007b; Lundh et al., 2013; Wolpert et al., 2008).

Procedures

Standard procedures at the clinics include adolescents and parents/carers to be invited to complete the user satisfaction measure, the ESQ, at either discharge or an evaluation point as part of clinical routines. At assessment, demographic data including age and gender was registered. There were no exclusion criteria.

Ethics

Gathering data in the Norwegian sample was approved by the Information Security Manager at UNN, who acts on behalf of the Norwegian Data Protection Authority. As the data was collected for the purpose of an audit, and only de-identified data was included in the analysis, no written consent was required from the families as procedures of anonymity and safe storage were followed. In the UK, as the study was a secondary analysis of anonymous routinely collected data sample, ethical approval was not required (NHS, 2023).

Statistical Analysis

Based on previous research, three competing models were tested with confirmatory factor analysis (CFA) using the mean- and variance-adjusted weighted least squares (WLSMV) estimator (Muthén & Muthén, 2018). The UK analyses were performed in RStudio using R version 4.0.3 (RStudio Team, 2020) and for the Norwegian samples we used R version 4.3.2, using the lavaan package (Rosseel, 2012). Every model was tested for parents/carers and adolescents in both the UK and Norway. Due to the large sample size, any participants who had not completed all the ESQ items were excluded from the UK analysis. In the Norwegian analysis, pairwise deletion was used. The main aim was to assess the most useful latent structure underlying the 12 items of the ESQ in the following competing models: (1) a unidimensional model where all 12 items load on a general factor of satisfaction (Attride-Stirling, 2002); (2) a two-factor model as suggested by Brown and colleges (2014) with factors for Satisfaction with Care (SWC) (items 1–7, and 11–12) and Satisfaction with Environment (SWE) (item 8–10); (3) a bifactor model with one general factor and two specific factors for SWC (items 1–7, and 11–12) and SWE (items 8–10).

The adequacy of the competing models was evaluated using a range of goodness-of-fit indices, including the Comparative FIT Index (CFI), Tucker-Lewis Index (TLI), root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR) and Chi-square ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\:{\upchi\:}}^{2}$$\end{document} ). Criteria were emphasized where CFI and TLI > 0.95, RMSEA < 0.06 (Hu & Bentler, 1999), and SRMR < 0.08 (Asparouhov & Muthén, 2018). Notably, the SRMR outperforms weighted root mean square residual (WRMR) in large samples (DiStefano et al., 2018). Additionally, the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\:{\upchi\:}}^{2}\:$$\end{document} is sensitive to very large samples (like we have for the UK sample), and even trivial misfit may be significant. Therefore, fit interpretations were not solely based on the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\:{\upchi\:}}^{2}$$\end{document} statistic.

The bifactor model was chosen as the superior model (see results below) based on the fit indices. To assess the reliability of the bifactor models, a series of indices were employed using Dueber’s online calculator (Dueber, 2017). As an index of unidimensionality, the Explained Common Variance (ECV) was used (Reise et al., 2010). Internal reliability for factors loading on the general factor was calculated by McDonald´s coefficient omega hierarchical (ω_h_), and McDonald’s omega specific (ω_hs_) was calculated to assess if items of the specific factors (Satisfaction with Care and Satisfaction with Environment) reliably explained residual variances. Higher values of ω_h_ and ω_hs_ indicate greater reliability (Reise et al., 2013). The ECV index represents the variance explained by the general dimension of the total common variance in the model. Notably, there is no “gold standard” ECV value to determine the question on uni-dimensionality (Reise et al., 2013). In a practical guide, Quinn (2014) suggests that an ECV above 0.90 points to a unitary construct being measured and for practical purposes supports the use of an overall score. Conversely, ECV values below 0.70 indicates sub-scores will provide added value over simply reporting an overall score of the construct measured. This suggests values that fall in the grey area between 0.70 and 0.90 need nuanced consideration (Quinn, 2014). In addition, Cronbach’s alpha (α) values were calculated to ascertain the internal consistency of the ESQ scale as a whole, and the two subscales. A value of 0.70 or higher was interpreted as having acceptable internal consistency (European Federation of Psychologists’ Associations, 2013).

Results

Descriptive Statistics

Norwegian Sample

The total number of participants in the Norwegian sample was 1205 patients. Following the exclusion of individuals with missing ESQ data, the final sample included ESQ responses from 380 parents/carers and 177 individual adolescents.

UK Sample

The overall UK sample consisted of 214,657 cases. Removing those without ESQ data, and any duplicate patient IDs resulted in a dataset comprising 9,761 parent/carer-reported ESQs and 10,207 adolescent-reported ESQs. The demographic characteristics of the participants included in the analyses are displayed in Table 1.

Table 1. Participant demographic variables at referralAdolescentParent/carerNorwayUKNorwayUK n

%

n

%

n

%

n

%

Gender Male (1)5631.6381737.421356.1487349.9Female (2)12168.4635362.216743.9488049.9 Missing 00370.410.38< 0.1 Ethnicity AsianNot available4224.1Not available4244.4Black3893.83463.5Mixed4274.24354.5Not stated112811.1104910.7Other1931.91982.0White695768.2660667.7 Missing 1771006916.83811007037.2Age of adolescent(mean/ SD)14.12.013.53.310.83.410.94.3 Missing 0012< 0.110.300N = 177 Norwegian adolescent sample, N = 10,207 UK adolescent sample, N = 380 Norwegian parent sample, N = 9761 UK parent sample

Confirmatory Factor Analysis

First, separate confirmatory factor analyses were conducted in each sample to evaluate the fit of the following models: (1) a unidimensional model; (2) a two-factor model with factors for Satisfaction with Care (items 1–7, and 11–12) and Satisfaction with Environment (item 8–10); (3) a bifactor model with one general factor and two specific factors for “Care” (items 1–7, and 11–12) and “Environment” (items 8–10). In three of the four samples, results for the bifactor model exceeded models 1 and 2, suggesting the bifactor model is superior to the alternative models. Details on results for the parent/carer and adolescent versions are presented separately for both Norway and the UK below.

CFA Parents/Carers

As seen in Table 2, for parents/carers in the Norwegian sample, the model fit considerably improved from model 1 to model 2, and from model 2 to model 3, indicating the Bifactor model had the most acceptable model fit for the Norwegian parent/carer version of the ESQ. For the parent/carer ESQ version in the UK sample, we found similar results to the parallel Norwegian sample, in terms of model fit improving from model 1 to model 2, and from model 2 to model 3 indicating the Bifactor model resulted in the best fit.

Table 2. Fit Indices, parent/carer versions of ESQModelChi-square (df; p)RMSEA (90% CI)CFITLISRMR Norway 1 Unidimensional327.9 (54; < 0.00005)0.116 (0.104; 0.128)0.9520.9410.1072 Two-factor187.9 (53; < 0.00005)0.082 (0.069; 0.095)0.9760.9700.0833 Bifactor88.4 (42; < 0.00005)0.054 (0.038; 0.070)0.9920.9870.051 UK 1 Unidimensional4954.0 (54; < 0.0001)0.112 (0.109; 0.114)0.9320.9160.0402 Two-factor4209.8 (53; < 0.0001)0.104 (0.101; 0.106)0.9420.9280.0313 Bifactor1615.7 (42 < 0.0001)0.072 (0.069; 0.075)0.9780.9650.019Norway N = 381, UK N = 7280. CFI = comparative fit index; TLI = Tucker-Lewis index; RMSEA = root-mean-square error of approximation; SRMR = standardized root mean square residual

While the RMSEA value slightly exceeded the ideal threshold of 0.06, the bifactor model showed the closest approximation compared to alternatives. Both CFI and TLI exceeded the recommended 0.95 threshold, indicating good fit. Additionally, the SRMR supported the bifactor model as the best fit, well below the common cut-off of 0.08. Despite marginal deviation in RMSEA, the bifactor model represented the data more adequately. The bifactor model yielded the most acceptable fit for the UK parent/carer version. Notably, significant chi-square values for both versions suggest misfit, partly due to large sample size in the UK. However, the magnitudes of RMSEA, SRMR, and CFI indicate misfit is not severe for the bifactor model, which was retained for further reliability testing.

CFA Adolescents

In the Norwegian adolescent sample, small sample size affected estimation (Table 3). Despite this, the unidimensional model (model 1) showed acceptable fit, suggesting limited benefit from adding complexity. Model 3 exhibited an R-square estimate for ESQ item number 1 larger than 1, rendering results unreliable for reliability testing. Non-significant chi-square values were observed for Norwegian adolescents in models 1 and 2. Assessing fit indices criteria, improvements were noted from model 1 to model 2, and model 2 to model 3 for UK adolescents (Table 3). The bifactor model emerged as the most suitable for this sample. Although chi-square values for UK adolescents were highly significant even in model 3, the large sample size partially contributed, with low RMSEA, SRMR, and high CFI indicating manageable misfit.

Table 3. Fit Indices, adolescent version of ESQModelChi-square (df; p)RMSEA (90% CI)CFITLISRMR Norway 1 Unidimensional58.7 (54; 0.31)0.022 (0.000; 0.054)0.9980.9980.0522 Two-factor51.3 (53; 0.54)0.000 (0.000; 0.045)1.0001.0010.046 UK 1 Unidimensional3148.0 (54; <0.0001)0.091 (0.088; 0.093)0.9390.9250.0392 Two-factor2497.7 (53; <0.0001)0.081 (0.079; 0.084)0.9520.9400.0303 Bifactor753.6 (42; <0.0001)0.049 (0.046; 0.052)0.9860.9780.019Norway N = 177, UK N = 6967. CFI = comparative fit index; TLI = Tucker-Lewis index; RMSEA = root-mean-square error of approximation; SRMR = standardized root mean square residual

To recap, the results from the separate CFAs indicate that the bifactor solution explains the data best in terms of model fit statistics in both countries for parent/carer ESQ and for adolescent ESQ in the UK. For the Norwegian adolescent ESQ, model fit was acceptable for model 1 where all items load on a unidimensional factor of satisfaction, but issues of a limited sample size must be considered. The two-factor solution found by Brown and colleagues (2014) also showed reasonable fit throughout the samples, except for a high RMSEA in the Norwegian parent/carer sample. As the bifactor model (Fig. 1) predominantly demonstrated best statistical model fit, this model was retained to proceed with reliability testing for three of the four samples.

Fig. 1. Bifactor model of the ESQ. SWC = Satisfaction with care; SWE = Satisfaction with Environment

Reliability

To substantiate the factor structure of the ESQ, we computed model-based reliability indices, including omega hierarchical ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} ) and omega specific ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} ) for each satisfaction factor. These indices, which offer a nuanced assessment of the reliability of the general and specific factors within the bifactor model, are detailed in Table 4. Table 4 shows that for UK parents/carers the general factor explained 88% of the common variance, with an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} of 0.62, suggesting that a substantial proportion of the variance is attributable to the general factor. In the same sample, the specific factor of Satisfaction with Care had only 3% explained common variance. For this subscale, an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} of 0.002 indicates that the large majority of reliable variance can be attributed to the general factor total score. In contrast, the specific factor Satisfaction with Environment accounted for a larger portion of the common variance at 9%, with a relatively low reliability for the total score for the subscale ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} = 0.11), capturing a modest amount of unique variance.

For Norwegian parents/carers, the general factor explained 78% of the common variance with an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} of 0.92, demonstrating strong reliability. For Norwegian parents/carers the specific factor Satisfaction with Care explained 9% of the common variance, with an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} of 0.01 indicating that, similarly to the UK sample, only a very small portion of residual variance of the Satisfaction with Care score is left after partitioning out the variability explained by the total score for the general factor. The specific factor of Satisfaction with Environment explained a larger portion of common variance with 13%, and an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} of 0.54, suggest that a significant portion of variance is uniquely attributable to this factor.

Similar results were found for the adolescent sample, with the general factor presenting acceptable reliability with most of the variance accounted for by the general factor. As displayed in Table 4, for UK adolescents, the general factor explained 87% of the common variance. The general factor total score accounted for 63% of the test score variance ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} = 0.63), which indicates that a relatively large portion of the reliable variance is accounted for by the general factor. In the same sample, the specific factor of Satisfaction with Care had only 5% explained common variance, with an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} of 0.01, while the factor Satisfaction with Environment explained 9% of the common variance with an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} of 0.12.

These results support the presence of a robust general satisfaction factor, with the Satisfaction with Environment subscale also reflecting some unique variance.

Table 4. Reliability indices of the ESQUKNorwayParent version ESQAdolescent version ESQParent version ESQ G

SWC

SWE

G

SWC

SWE

G

SWC

SWE ECV0.8760.0330.0920.8670.0460.0870.7760.0940.130 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{{\upvarpi\:}}_{\text{h}}/{{\upvarpi\:}}_{\text{h}\text{s}}$$\end{document} 0.6190.0020.1140.6310.0100.1170.9210.0080.538α0.940.950.680.920.930.660.880.900.60ECV = explained common variance; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} = reliability for the general factor test score; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} = the reliability of the specific scores controlling for the general score, based on items relevant to the specific factor; alpha = Cronbach’s alpha; G = General factor; SWC = Satisfaction with Care; SWE = Satisfaction with Environment

Discussion

In the current study, we assessed the factor structure and psychometric properties of the Experience of Service Questionnaire (ESQ) in Child and Adolescent Mental Health Services (CAMHS) in Norway and the UK, aiming to determine its validity and reliability, particularly examining its factor structure and subscale distinctiveness.

In the confirmatory factor analysis, the bifactor model demonstrated superior fit (according to fit indices as diverse as the CFI, RMSEA and the SRMR) compared to both the unidimensional model and the two-factor model proposed by Brown et al. (2014), which includes the factors Satisfaction with Care (SWC) and Satisfaction with Environment (SWE). Kotsis et al. (2024) recently found support for a unidimensional structure in the Greek version of the ESQ, suggesting that cultural or methodological differences may influence the factor structure across different populations. In addition, Reise et al. (2013) noted that the presence of multidimensionality does not ensure that subscales will yield meaningful and reliable information distinct from the general factor. This is evident in our study, where the SWC subscale score contributes minimal reliable information beyond the general satisfaction score across all three samples examined.

Interestingly, while the SWC subscale, comprising 9 of the 12 items, showed low unique reliability, the SWE subscale displayed higher omega hierarchical values, particularly in the Norwegian parent sample. The three items of the SWE subscale pertaining to the treatment facilities, appointment scheduling, and clinic accessibility may be seen as complex in their measurement of satisfaction. These items could be considered formative indicators of an environmental construct, as they do not necessarily reflect a latent trait of the respondent. Nonetheless, respondents do apply their personal judgment to these aspects of the treatment process, suggesting that a reflective measurement model may still be appropriate.

Given these findings, the utility of the SWC subscale as a distinct measure is questionable. Future research should consider whether modifications to the SWC items could enhance their ability to capture unique variance, or whether alternative methods of assessing specific aspects of care satisfaction are needed.

The results align with a growing body of literature emphasizing the importance of user satisfaction assessment in CAMHS (Athay & Bickman, 2012; Ayton et al., 2007; Nicholas et al., 2017; Rickwood et al., 2017; Simmons et al., 2014). Notably, this study builds on prior research (Attride-Stirling, 2002; Brown et al., 2014; Bunge et al., 2014; Davison et al., 2017; Kotsis et al., 2024) thereby adding insights to the cross-cultural applicability of the ESQ and its validity across diverse clinical settings.

The recognition of a general factor adds depth to the comprehension of user satisfaction in CAMHS. This general factor of user satisfaction suggests an underlying, fundamental dimension contributing to the overall perception of satisfaction, encompassing beyond individual components, that likely considers diverse aspects working together to create a satisfactory experience at CAMHS.

Moreover, identifying a general factor of user satisfaction within CAMHS mirrors already established conceptualizations in psychological research, such as the g-factor of intelligence (Deary et al., 2010; Jensen, 1998) and the p-factor of psychopathology (Caspi et al., 2014; Murray et al., 2016; Patalay et al., 2015). The recognition of this common factor not only aligns with established psychological constructs, but this conceptual alignment also provides a theoretical framework for understanding user satisfaction, fostering a nuanced understanding of the complexities inherent in the evaluation of care in CAMHS.

While highlighting the general factor of user satisfaction took precedence in this study, the specific factors of Satisfaction with Care (SWC) and Satisfaction with Environment (SWE) merit consideration as well. It is worth noting that specific factors were residualized in the bifactor model, indicating what remains after accounting for general satisfaction. However, the very low \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs}$$\end{document} values for the SWC, suggest that its total score does not capture much unique information beyond what is already explained by the general factor of satisfaction total score. Therefore, the SWC total score seems largely redundant since the general satisfaction factor already accounts for most of the variance in the scores. This redundancy implies that the SWC score may not provide additional insight into the specific aspects of care that are distinct from overall satisfaction.

While interpreting hierarchical omega values, caution is warranted as they serve as an index reflecting the proportion of reliable systematic variance of a subscale after residualization (Reise et al., 2013), and should not be construed as a measure of “reliability” in the traditional sense (Rodriguez et al., 2016). However, our results underscore the unique contribution of the SWE subscale to the overall reliability of the ESQ. Therefore, we advise conservative interpretations, acknowledging that the specific factors may not represent distinct dimensions beyond the general factor. Nonetheless, it is noteworthy that specific factors, especially the SWE subscale, still hold inherent value in assessing user satisfaction in CAMHS, particularly concerning the clinical implications of user satisfaction measures. While our discussion highlights the importance of the “Satisfaction with Environment” (SWE) subscale in enhancing the ESQ’s overall reliability, it’s important to note the predominant emphasis on the Care aspect, evident from the loading of the two general satisfaction items onto the Care subscale. The structural and content differences between the SWC and SWE indicators may contribute to the perceived unique value of the SWE subscale. However, caution is needed in ascribing inherent value to the SWE subscale, as it may also reflect respondent characteristics or subjective perceptions rather than solely objective service quality factors. While these items capture important aspects of service accessibility and logistics, their interpretation may vary depending on individual expectations and contextual influences. Nevertheless, the SWE subscale remains valuable in assessing user satisfaction in CAMHS, particularly concerning its clinical implications.i Additionally, our study found high Cronbach’s alpha values for the general factor in both the UK parent (α = 0.94) and adolescent (α = 0.92) samples, suggesting strong internal consistency under the assumption of unidimensionality. However, the omega hierarchical values were notably lower ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} = 0.62 for parents and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{h}$$\end{document} = 0.63 for adolescents), indicating that the general factor alone does not account for the majority of the variance in the observed scores. This discrepancy highlights the presence of multidimensionality within the scale, and it suggests that specific factors also contribute significantly to the scale’s structure. Therefore, while the scale items are highly interrelated, the interpretation of the general factor should be made with an understanding that it does not fully capture the complexity of user satisfaction as measured by the ESQ.

As the reliability analyses revealed consistent internal consistency for the general factor across both the Norwegian and UK parent/carer, and UK adolescent samples, this suggests the general factor effectively captures the essence of user satisfaction common to both cultural contexts. As robust psychometric properties were revealed in both the Norwegian and UK samples the overall structure and validity of the ESQ appear to transcend cultural boundaries. This suggests the ESQ could serve as a valid measure for assessing user satisfaction in diverse cultural settings.

The implications of our findings for clinical practice in CAMHS are significant. The present study recognizes the multi-faceted nature of user satisfaction, establishing the general factor in user satisfaction as well as specific factors which necessitate targeted interventions but also underscores the interconnectedness of those factors. Especially, the results of our reliability analysis revealed the unique contribution of the subscale SWE to the overall reliability of the ESQ. The omega hierarchical coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs})$$\end{document} linked to this subscale indicate that a significant amount of reliable variance persists even after factoring in the general factor. Conversely, the results imply SWC subscale may not capture a distinct dimension beyond the general factor, as indicated by the omega specific coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\varpi\:}_{hs})$$\end{document} . As a result, to enhance user satisfaction in CAMHS targeted interventions must be carefully tailored to address specific contexts, with particular emphasis on factors such as the physical environment, accessibility, and scheduling of appointments, which all play pivotal roles in user satisfaction in CAMHS.

Tailoring care interventions to address communication dynamics between clinicians and users, particularly adolescents and their parents, holds promise for enhancing satisfaction. Moreover, acknowledging the impact of the service environment on user experience underscores the need for organizational enhancements within CAMHS. Given the presence of a general satisfaction factor, prioritizing quality improvement emerges as a priority. By combining efforts to address specific concerns with an overarching focus on improving overall satisfaction, CAMHS can make comprehensive advancements. Notably, efforts to enhance satisfaction also play a pivotal role in reducing premature termination of treatment and fostering positive relationships between families and clinicians (Bjørngaard et al., 2008; Hawley & Weisz, 2005). Therefore, routine assessment user satisfaction, encompassing both general satisfaction and specific dimensions, is recommended to ensure that CAMHS continually evolves to meet the diverse needs of its users. In summary, our study not only refines our understanding of user satisfaction in CAMHS but also provides actionable insights for clinicians and service providers to elevate the quality of care and overall user experience.

Acknowledging the strengths of our study, certain limitations must also be considered. Regarding generalizability, certain limitations need consideration. First, the relatively small sample size in the Norwegian adolescent cohort poses challenges to the generalizability of findings. Additionally, we need to exercise caution regarding ethnicity. The UK sample is predominantly White, while ethnicity data is lacking for the Norwegian sample. This skew in representation restricts the generalizability of findings to more diverse populations, meaning caution must be taken when extrapolating conclusions beyond this demographic subset. Additionally, the significant chi-square values in the UK sample warrant cautious interpretation, given the influence of large sample sizes on statistical significance.

Future research endeavours should further examine the ESQ´s applicability across cultural contexts. Since Kotsis et al. (2024) identifyed a unidimensional factor structure in the Greek ESQ, future studies should investigate whether differences in service delivery models, translation procedures, or cultural expectations shape satisfaction ratings. Furthermore, measurement invariance analyses could determine whether the observed differences in factor structures reflect genuine cross-cultural variations or methodological discrepancies in questionnaire adaptation and administration. Also, the limited unique reliability observed in the subscale SWC might benefit from further investigation and possible refinement to better capture the nuanced aspects of satisfaction with care, while enhancing the sensitivity and specificity of the ESQ in assessing user experiences in CAMHS. Additionally, investigations into the responsiveness of the ESQ to specific interventions or changes in service delivery could provide valuable insights for ongoing quality improvement initiatives in CAMHS.

Conclusion

In summary, this study presents nuanced findings from a psychometric evaluation of the treatment satisfaction measure Experience of Service Questionnaire. A bifactor model demonstrated good model fit, suggesting that a general satisfaction factor underlies the various aspects of treatment satisfaction. However, the analysis also revealed that the Satisfaction with Care (SWC) subscale, which was initially designed to capture specific elements of patient care, exhibited minimal reliable variance when the general factor was accounted for. This indicates that the SWC subscale may not be measuring a distinct dimension of satisfaction beyond what is captured by the general factor.

Moreover, the unexpected high correlations among items within the Satisfaction with Environment (SWE) subscale, particularly those related to the facilitation of treatment, suggest that general respondent satisfaction, potentially influenced by broader life satisfaction or mood, may be confounding the interpretation of the SWE subscale scores. This pattern points to an overarching influence of what we may call respondent disposition, complicating the attribution of satisfaction scores solely to the treatment environment.

These findings underscore the complexity of interpreting subscale scores in the context of treatment satisfaction and highlight the importance of considering general respondent satisfaction when evaluating specific aspects of patient care. The study calls for a careful examination of the factors that influence satisfaction measures and suggests that future research should aim to disentangle the specific contributions of treatment-related factors from the broader psychological state of respondents.