A proxy measure of clinical insight in psychosis: an electronic health records-based validation study

TL;DR

This study developed a reliable proxy measure for assessing clinical insight in psychosis using electronic health records data.

Contribution

A novel proxy measure for insight in psychosis validated using EHR data and the SUMD scale.

Findings

The proxy measure showed strong criterion validity with SUMD scores (r = 0.61, P < 0.001).

The proxy predicted good insight status with high odds (OR = 20.95, 95% CI 7.32–59.91).

The proxy demonstrated excellent interrater reliability (ICC = 0.88).

Abstract

Insight assessment in psychosis remains challenging in practice-oriented research.

To develop and validate a proxy measure for insight based on information from electronic health records (EHR). For that purpose, we used data on the Scale to Assess Unawareness of Mental Disorder (SUMD) and data from EHR notes of patients in an early psychosis intervention programme (Programa de Atención a Fases Iniciales de Psicosis, Santander, Spain).

Junior and senior clinicians examined 134 clinical notes from 106 patients to explore criterion and content validity between SUMD and a clinician-rated proxy measure, using three SUMD items.

In terms of criterion validity, SUMD scores correlated with the proxy (r = 0.61, P < 0.001), even after adjusting for the following confounders: type of psychotic disorder, clinical remission status and rater experience (r = 0.58, P < 0.001); and the proxy predicted good insight status (odds ratio 20.95, 95% CI 7.32–59.91, P < 0.001). Regarding content validity, the three main SUMD subscores correlated with the proxy (r = 0.55–0.60, P < 0.005). There were no significant differences in age, gender or other clinical variables, i.e. discriminant validity, and the proxy significantly correlated with validated psychometric instruments, i.e. external validity. Intraclass correlation coefficient (i.e. interrater reliability) was 0.88 (95% CI 0.59–1.00, P < 0.05).

This SUMD-based proxy measure was shown to have good to excellent validity and reliability, which may offer a reliable and efficient alternative for assessing insight in real-world clinical practice, EHR-based research and management. Future studies should explore its applicability across different healthcare contexts and its potential for automation, using natural language-processing techniques.

Figures

Scale to Assess Unawareness of Mental Disorder (SUMD) proxy-based assessment: examiner’s guide.

Examples of clinical notes and Scale to Assess Unawareness of Mental Disorder (SUMD) proxy scoring method.

Full text

A multicentre study conducted by the World Health Organization in 1973 demonstrated that the majority of individuals diagnosed with schizophrenia exhibited a marked lack of insight,^1^ which was subsequently replicated.^2^ Impaired insight has thus become a core feature of psychosis from early phases and is associated with premorbid personality traits.^3^ The multidimensional models of insight – illness awareness, symptom relabelling, treatment compliance and awareness of the social consequences, proposed in the early 1990s^7,8^ – led to three decades of extensive research in the area.^9^ Thanks to this work, insight in psychosis has been consistently linked to positive outcomes – greater insight and better outcomes, namely reduced psychotic symptom severity,^10^ reduced use of coercive treatments^9^ and better psychosocial functioning.^11,12^ More controversially, insight gain may result in depression,^10,13^ increased risk of suicidal behaviour^14–17^ and poorer self-perceived quality of life.^18^ Although so-called insight paradox^19^ remains far from clear,^20^ the aforementioned strong association of insight with positive outcomes results in insight assessment becoming a matter of major clinical relevance.^21^

Insight measurement can be challenging in both routine clinical practice and research, especially given its conceptual complexity. The first insight assessments relied on patients’ narrative accounts of their mental health issues, which could not be used for research purposes. To address this issue, investigators designed interviews that classified patients as having ‘good’ or ‘poor’ insight: for instance, the Present State Examination.^22^ Subsequently, semi-structured scored interviews were validated to measure insight unidimensional, such as the Insight and Treatment Attitudes Questionnaire (ITAQ).^23^ Finally, in order to capture the aforementioned multidimensionality of insight,^7,8^ two scales were validated, namely the Scale to Assess Unawareness of Mental Disorder (SUMD)^24^ and its more commonly used shortened version,^2^ and the Schedule for Assessment of Insight (SAI),^8^ which also had an expanded version (SAI-E).^25^ Further insight assessments include self-reports such as the Birchwood Insight Scale,^26^ the Markova and Berrios Insight Scale (MBIS)^27^ and the specific insight item of two large scales of general psychopathology, the Positive and Negative Symptoms Scale (PANSS)^28^ and the Manual for the Assessment and Documentation of Psychopathology (AMDP system).^29^ However, these insight assessments require considerable time input and must be rated by trained staff.^30^ Moreover, some degree of cooperation is required that may introduce a selection bias into research studies, namely the exclusion of potentially eligible participants with poor insight, especially in randomised controlled trials.^31^

These limitations of previous approaches to insight measurement are more pronounced in real-world epidemiological studies, given the scant incorporation of psychometric tests into daily clinical practice.^30^ Alternatively, insight in psychosis (hereafter referred to as simply insight) could be assessed in time-constrained clinical settings with proxy measures, which can be defined as ‘indirect measures of the outcome of interest (i.e. insight), with which there is strong correlation’. Proxy measures are commonly used when the outcome variable cannot be directly observed/assessed,^32^ and have had their utility proven in mental health science research. A variety of proxy measures have been developed and validated;^33,34^ those derived from either sociodemographic variables or unstructured electronic health records (EHR) data have proved particularly useful in practice-oriented research studies.^35,36^ Nevertheless, the widespread implementation of proxy measures in routine clinical care and EHR-based research remains a pending subject.

Aims

The objective of this study was to assess the validity and reliability of a SUMD-based proxy measure for insight assessment in psychosis using EHR. Specifically, we examined the extent to which a three-level clinician rating of insight, derived from routine EHR notes, aligns with SUMD scores in subjects with psychosis. If validated, this approach could provide a valuable tool for both routine clinical practice and real-world EHR-based research.

Method

Study population and inclusion/exclusion criteria

Data for this study were retrieved from the 3-year follow-up, first-episode psychosis (FEP) Programa de Atención a Fases Iniciales de Psicosis (PAFIP) cohort, a publicly funded early intervention service (EIS) for people with psychosis, at Marqués de Valdecilla University Hospital (Santander, Spain). A detailed description of PAFIP is available elsewhere.^37,38^ Briefly, referrals to PAFIP came from local/regional in- and out-patient mental health services, emergency departments and primary care trusts in Cantabria (on the northern coast of Spain, with a catchment area population estimated at approximately 535 000). Potentially eligible candidates were screened against the following inclusion criteria: (a) age between 16 and 60 years, both inclusive; (b) living in the catchment area; (c) experiencing a FEP as outlined below; (d) meeting DSM-IV criteria for a non-affective psychotic disorder, which were confirmed by the Structured Clinical Interview for the DSM; and (e) no previous treatment with antipsychotic medication or, if previously treated, a total treatment duration of less than 6 weeks. Drug dependence (excluding nicotine), having an IQ <70 according to the Wechsler Adult Intelligent Scale-III vocabulary subtest and a history of neurological disease or head injury were exclusion criteria. The project was approved by the local research ethics committee (CEIm Cantabria; internal code of approval, no. 2014.245) in accordance with international standards for research ethics. Participants provided written informed consent as approved by the local research ethics committee. The characteristics of the PAFIP cohort were reported to be generalisable to the target population.^38^ This study was reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement guidelines.^39^

Selected demographic, clinical and psychometric variables

Data on demographic variables, premorbid characteristics and clinical status were collected from face-to-face interviews with patients, key informants and medical records at the time of PAFIP inception. Sociodemographic variables included age, gender, ethnicity, marital or cohabitation status, employment status, educational level, urbanicity status (classified as living in a municipality with more than 10 000 inhabitants), socioeconomic status and living arrangements at the onset of psychosis. Clinical characteristics included family history of psychosis, history of substance use (defined as harmful or hazardous use of alcohol, cannabis or other illicit drugs), premorbid adjustment, premorbid IQ, age at onset of psychosis, duration of untreated psychosis (DUP), disease duration (from the onset of psychosis until clinical records were examined for SUMD proxy scoring), level of psychosocial functioning and neurocognitive performance and type of psychotic disorder (i.e. DSM-IV coding).

Socioeconomic status was assessed using the Hollingshead and Redlich scale. Premorbid psychosocial adjustment was measured using the Cannon–Spoor Premorbid Adjustment Scale (PAS), which, for the sake of interpretation, was dichotomised into good/poor premorbid adjustment through the median. The Wechsler Adult Intelligence Scale (third edition, WAIS-III) Vocabulary Subtest estimated premorbid IQ. Social functioning was assessed with the Disability Assessment Schedule (DAS). The level of global cognitive functioning (GCF) provided an overall measure of neurocognitive performance.

PAFIP full assessments were carried out at baseline, 6 weeks, 12 months and 36 months, in accordance with the PAFIP protocol. In addition, follow-up clinical appointments were arranged on a monthly basis over the follow-up. Psychopathological symptoms were assessed using the expanded version of the Brief Psychiatric Rating Scale (BPRS), the Scale for the Assessment of Positive Symptoms (SAPS) and the Scale for the Assessment of Negative Symptoms (SANS). Symptomatic remission was defined according to the criteria set out by Andreasen et al.^40^

Insight assessment

Lack of insight was measured with the shortened version of SUMD. Specifically, the three general items of the shortened version of SUMD – (a) awareness of having a mental disorder, (b) awareness of the need for treatment and (c) awareness of the social consequences of the disorder – were considered. Scores for each item ranged from 1 to 5, with higher scores indicating poorer insight, which were summed to create total SUMD scores ranging from 3 to 15. Those with a SUMD total score of 6 or below had ‘good insight’, while a total score of 7 or above indicated ‘poor insight’.^41^ SUMD has been widely used in both clinics and research, and was demonstrated to have good reliability and validity.^41,42^

Proxy measures for insight assessment from SUMD

The proxy version of SUMD is shown in Fig. 1, according to which, total insight scores were classified unidimensional on a 3-point Likert scale as either good (score of 3), partial (score of 9) or poor insight (score of 15). To rate the SUMD proxy, clinicians carefully read the patient’s EHR and considered their clinical status at that point (acutely psychotic or clinically stable). Where no insight assessment was documented (which is relatively common in routine clinical practice due to time constraints during visits), raters considered patient psychopathological/behavioural information, as well as clinical setting (in-/out-patient) and their clinical experience and understanding of psychotic disorders, to provide scores on the SUMD proxies. Some representative examples of the scoring method are shown in Fig. 2.

Fig. 1. Scale to Assess Unawareness of Mental Disorder (SUMD) proxy-based assessment: examiner’s guide.

Fig. 2. Examples of clinical notes and Scale to Assess Unawareness of Mental Disorder (SUMD) proxy scoring method.

Standardised procedure for validation of the SUMD proxy measure

For study purposes, the proxy version of SUMD was validated using EHR and psychometric assessments from the 1- and 3-year follow-up appointments of the PAFIP programme. Raters were trained in the use of the proxy tool, and followed a specific guide designed by the first author (L.-D.A.). Raters’ experience (less than or over 5 years) was recorded and tested as a potential moderator of proxy scoring. Cases were selected if: (a) EHR were available at the time of the 1- and 3-year follow-up PAFIP psychometric assessments; (b) the clinician recording the patient’s mental state (i.e. EHR) was not the evaluator who conducted the psychometric assessment; (c) the SUMD score was unavailable in the EHR; and (d) the third clinician evaluating the EHR to rate the proxy measure of insight was blinded to the SUMD scores. If an EHR lacked sufficient information pertaining to the proxy insight assessment (e.g. ‘no changes from previous visit’), the closest valid EHR (within a 1-month time frame) to the SUMD assessment was used instead. In order to evaluate the psychometric validity of the proxy measures, the degree of concordance between SUMD and proxy scores from EHR was examined using a variety of statistical methods detailed below.

Data analysis

Descriptive statistics were used to examine the demographic and clinical characteristics of the sample, which are reported as either percentages, mean or median and standard deviation or interquartile range (IQR), as appropriate. Test statistics of the proxy measure were specifically assessed using four types of psychometric validity: criterion, content, discriminant and external validity, and interrater reliability.

In order to test the criterion validity (i.e. the extent to which the proxy measure was correlated with SUMD scores), both the concurrent and predictive validity were examined. First, unadjusted univariate correlation analyses between proxy SUMD and SUMD scores were performed and reported as Pearson’s or Spearman’s coefficient, depending on whether the variable followed a normal distribution, respectively, according to the Kolmogorov–Smirnov normality test. Second, partial correlations were run between proxy SUMD and SUMD scores while controlling for potential confounders, such as type of psychotic disorder (first-episode schizophrenia versus all other psychotic disorder diagnoses), clinical status (remission versus non-remission status according to Andreasen’s criteria) and the rater’s previous experience in scoring proxies (less than or more than 5 years). Finally, the extent to which the proxy SUMD predicted those cases with good/poor insight was determined by a binary logistic regression analysis (using the above cut-off points). Odds ratios and the corresponding 95% confidence intervals were estimated. Nagelkerke’s R^2^ was used to measure the goodness of fit of the resulting model, and the area under the receiver operating characteristic (ROC) curve (AUC) evaluated the classifier’s performance of the proxy.

For content validity (i.e. the extent to which the proxy assessed and represented all facets of the abbreviated SUMD construct), the correlation coefficients between proxy SUMD scores and those for the three main insight domains of the SUMD were examined. Discriminant validity (i.e. the extent to which the proxy measure was not related to other clinical variables or different constructs) was determined by comparing proxy SUMD median scores across patients’ age, gender, educational level, living status and illness duration. External validity (i.e. the extent to which the results of the proxy tool can be generalised or transferred to other psychometric tests) was assessed by bivariate correlations between proxy SUMD scores and BPRS, SAPS, SANS and DAS scores.

The interrater reliability of the proxy method was evaluated using the intraclass correlation coefficient (ICC), a two-way random-effects model and absolute agreement, which was calculated from three clinical notes randomly selected and rated by six researchers from all participating centres.

The handling of missing data was conducted in accordance with the pairwise deletion method. The level of significance was set at P < 0.05. Post hoc power analyses were performed using G-Power software (version 3.1.9.4, Franz Faul, University of Kiel, Germany), assuming a medium effect size (P = 0.3) and α = 0.05. The remaining statistical analyses were conducted with IBM SPSS Statistics software, version 29 for MacOS.

Results

Sample characteristics

A total of N = 134 clinical notes from n = 106 patients from the PAFIP programme were included. The majority of the clinical notes (57.5%, n = 77) were from the PAFIP 1-year follow-up appointment, 46.3% (n = 62) were from first-episode schizophrenia cases and 59% (n = 79) were from patients in remission. Seventy-six (56.7%) EHR notes were assessed by experienced clinicians (i.e. with >5 years of clinical experience) for proxy insight scoring. Sociodemographic and clinical features of the cases analysed are shown in Table 1.

Table 1. Sociodemographic, clinical and psychometric characteristics of the cases analysedCharacteristicsTotal sample(N = 134)^a^ Sociodemographics  Age (years), median (IQR)34.57 (23.24–42.60) Gender (male), n (%)71 (53) Ethnicity (European White), n (%)117 (87.3) Marital status (unmarried and without a partner), n (%)87 (64.9) Employment status (unemployed and not studying), n (%)42 (31.3) Education (secondary or lower education), n (%)110 (82.1) Socioeconomic status of parents (low), n (%)76 (56.7) Urban area, n (%)96 (71.6) Living arrangements (alone), n (%)14 (10.4) Clinical features  Family history of psychosis, n (%)34 (25.4) Substance use (yes), n (%)76 (56.7) Premorbid adjustment (poor)^b^, n (%)68 (50.7) Premorbid IQ, median (IQR)100 (90.00–106.25) Age at onset (years), median (IQR)32.34 (22.83–41.81) Duration of untreated psychosis (months)3 (0.50–10.00) Disease duration (years), median (IQR)3 (1.12–3.3) Level of social functioning (DAS), median (IQR)0 (0–2) Global Cognitive Functioning, median (IQR)0.64 (0.18–1.25) Type of psychotic disorder (first-episode schizophrenia), n (%)62 (46.3) Psychotic symptoms characteristics  Brief Psychiatric Rating Scale (BPRS) total score, median (IQR)26 (24–33) Scale for the Assessment of Positive Symptoms (SAPS) total score, median (IQR)0 (0–0) Scale for the Assessment of Negative Symptoms (SANS) total score, median (IQR)2 (0–8) Symptomatic remission (Andreasen criteria), n (%)79 (59) Scale to Assess Unawareness of Mental Disorder (SUMD), median (IQR)3 (3–7)DAS, Disability Assessment Schedule; IQR, interquartile range.a. A total of 134 clinical notes from 106 patients were included in the analyses.b. The median score (2.05) on the Premorbid Adjustment Scale (PAS) was used as the cut-off point for determination of good premorbid adjustment.

The psychometric characteristics of the sample are summarised in Table 1. The median SUMD total score was 3 (IQR 3–7) and the median SUMD proxy score was 3 (IQR 3–9). In terms of good insight categorisation, 71.6% (n = 96) of patients assessed with SUMD were classified as having good insight compared with 58.2% (n = 78) who met the proxy criteria. The post hoc power analysis (1 – b error probability) yielded a value of 0.95, indicating that the sample size had sufficient statistical power to detect significant associations between SUMD and its proxy version.

Criterion validity

SUMD score significantly correlated with its proxy (r = 0.61, 95% CI 0.48–0.71, P < 0.001). After controlling for confounders, the partial rank correlation coefficient result was similar (r = 0.58, 95% CI 0.45–0.68, P < 0.001), thus indicating good concurrent validity of the proxy tool. Regression analysis showed that the proxy SUMD predicted good insight status (odds ratio 20.95, 95% CI 7.32–59.91, P < 0.001), with the model accounting for 42% of the variance on insight status. In total, 79.1% of cases were correctly classified in terms of good/poor insight. The AUC was 0.76 (95% CI 0.67–0.85, P < 0.001), indicating that the proxy SUMD had accurate predictive validity.

Content and discriminant and external validity

Correlation coefficients between the SUMD proxy and each of the three SUMD domains (awareness of mental disorder, awareness of need for treatment and awareness of the social consequences) ranged from r = 0.55 to r = 0.60 (all of which were significant at P < 0.001), indicating good content validity. No significant differences in age, gender, educational level, living arrangement or disease duration were observed, demonstrating excellent discriminant validity of the proxy version. SUMD proxy scores showed low to medium significant correlations with BPRS (r = 0.33, 95% CI 0.17–0.48, P < 0.001), SAPS (r = 0.49, 95% CI 0.34–0.61, P < 0.001), SANS (r = 0.17, 95% CI 0–0.34, P < 0.047) and DAS scores (r = 0.21, 95% CI 0.03–0.37, P < 0.018), indicating optimal external validity of the proxy tool.

Interrater reliability

The ICC of the SUMD proxy was 0.88 (95% CI 0.59–1.00, P < 0.001), which indicated good interrater reliability. The psychometric validity results are detailed in Table 2.

Table 2. Summary of SUMD scores and psychometric properties of its proxy versionSummariesScores Summary of SUMD assessments  G1: insight into having a mental disorder, median (IQR)3 (1–3) G2: insight into the need for treatment, median (IQR)3 (1–2) G3: insight into the social consequences of the disorder, median (IQR)3 (1–2) SUMD total score (G1 + G2 + G3), median (IQR)3 (3–7) Percentage of cases categorised as ‘good insight’, n (%)96 (71.6) Summary of proxy SUMD assessments  Proxy SUMD scores, median (IQR)3 (3–9) Percentage of cases categorised as ‘good insight’, n (%)78 (58.2) Summary of the psychometric validity of proxy SUMD

* Criterion validity*   Spearman correlation coefficient with SUMD total score, rho (95% CI)0.61 (0.48–0.71)*  Partial rank correlation coefficient with SUMD total score (95% CI)0.58 (0.45–0.68)*  Predictive ability for SUMD ‘good insight’ classification, OR (95% CI)20.95 (7.32–59.91)*  Nagelkerke R^2^ of the prediction model0.42  Overall percentage of SUMD ‘good insight’ correct predictions (%)79.1  AUC of the model (95% CI)0.76 (0.67–0.85)* * Content validity*   Insight into having a mental disorder, rho (95% CI)0.60 (0.47–0.70)*  Insight into the need for treatment, rho (95% CI)0.55 (0.42–0.66)*  Insight into social consequences of the disorder, rho (95% CI)0.58 (0.46–0.69)* * Discriminant validity*   Age, rho (95% CI)0.28 (−0.15 to 0.20)^NS^  Gender, RPB (95% CI)−0.01 (−0.18 to 0.17)^NS^  Educational level, RPB (95% CI)0.12 (−0.05 to 0.29)^NS^  Living arrangement, RPB (95% CI)0.03 (−0.14 to 0.21)^NS^  Disease duration, rho (95% CI)−0.26 (−0.21 to 0.16)^NS^ * External validity*   Brief Psychiatric Rating Scale (BPRS), rho (95% CI)0.33 (0.17–0.48)*  Scale for the Assessment of Positive Symptoms (SAPS), rho (95% CI)0.49 (0.34–0.61)*  Scale for the Assessment of Negative Symptoms (SANS), rho (95% CI)0.17 (0–0.34)*  Disability Assessment Schedule (DAS), rho (95% CI)0.21 (0.03–0.37)* * Interrater reliability*   Intraclass correlation coefficient, ICC (95% CI)0.88 (0.59–1.00)AUC, area under the curve; OR, odds ratio; IQR, interquartile range; NS, not significant; RPB, point biserial correlation; SUMD, Scale to Assess Unawareness of Mental Disorder.*P < 0.05.

Discussion

Main findings

This study aimed to assess the validity and reliability of a SUMD-based proxy measure for insight assessment in psychosis using EHR. To this end, EHR and psychometric assessments from a 3-year follow-up FEP cohort of patients under an EIS (PAFIP programme) from Santander (Spain) were used. In order to evaluate the validity of this proxy tool, the concordance between SUMD and its proxy version scores was examined while controlling for the effects of other explanatory variables, including the type of psychotic disorder (first-episode schizophrenia versus all other diagnoses), in-/out-patient status and raters’ experience. Of relevance, the SUMD-based proxy measure was demonstrated to have robust construct and content validity with SUMD, along with excellent discriminant validity against other unrelated constructs and optimal external validity in relation to other validated instruments for assessment of symptom severity and disability. To the best of our knowledge, this is the first study to validate an EHR-based proxy tool for the assessment of insight in patients with psychotic disorders under real-world conditions.

Overall performance of this insight proxy measure, including four types of validity – criterion, content, discriminant and external validity, and interrater reliability – was found to be good to excellent. In terms of construct validity, the psychometric properties of the SUMD proxy fulfilled quality standards for the validation of proxy tools.^32^ When compared with other validated clinician-rated proxy measures in the field of psychosis, the correlation coefficient for the proxy SUMD (r = 0.61) was slightly higher than in a previous validation study of a proxy measure of PAS (r = 0.57).^35^ Of note, this proxy version of SUMD correctly classified more than 3 of 4 individuals in terms of good/poor insight status (79.1%), in line with external validation of the PANSS-6, Young Mania Rating Scale 6 (YMRS-6) and Montgomery–Asberg Depression Rating Scale 6 (MADRS-6) proxies, which ranged from 76.7 to 83.3%.^36^ The AUC for the proxy SUMD (0.76) demonstrated slightly enhanced discriminative power compared with a proxy measure of the PANSS remission criteria based on Clinical Global Impressions Scale (CGI) scores (0.73).^43^ In terms of interrater reliability, the SUMD proxy method ICC (0.88) was higher than that in a previous validation study of proxy versions of PANSS-6, YMRS-6 and MADRS-6.^36^ Finally, outside the framework of clinician-rated proxy tools but within the realm of third-party proxy tools, it should be noted that the EHR-based proxy SUMD had a stronger correlation (r = 0.61) with SUMD than the validated caregivers’ version of this scale (r^i^ = 0.48).^44^ Above and beyond significant differences between clinician- and caregiver-rated scales,^45^ this finding suggests that the proxy SUMD version may provide a more accurate estimate of patient insight from review of EHR than the daily impressions of their close relatives or key informants. Taken together, these comparisons appear to indicate that the proxy SUMD may become a valid and reliable instrument for measurement of clinical insight in psychotic disorders, in both daily practice and EHR-based research.

A novel contribution to insight assessment in psychosis

In 1934, Sir Aubrey Lewis characterised insight as ‘a correct attitude to a morbid change in oneself’, emphasising that it could be inferred only from the patient’s behaviour, particularly their verbal expressions.^46^ Based on the multidimensional models of insight proposed in the 1990s,^7,8^ scales measuring multiple insight dimensions, such as SUMD and the SAI-E, were validated.^24,25^ In spite of the conceptual complexity involved, these validation studies demonstrated that insight can be measured, especially given the good level of agreement between these scales.^21,47^ In today’s mental health crisis, however, clinicians could be tempted to avoid time-related issues, hence assessing insight in an unidimensional manner as present/absent, which cannot generate data for research purposes and may prevent them from monitoring potential insight changes over time. By building on previous validation studies of EHR-based proxy measures in psychosis,^35,36^ the present study developed and validated a SUMD-based proxy tool for the assessment of insight in psychosis, which provides an overall measure of patients’ insight level from EHR. This clinical insight information could be considered a good predictor of patient outcomes, and may provide valuable information for early intervention and treatment planning.

Furthermore, as alluded to by Lewis^46^, insight cannot be measured entirely objectively like a biological measure which relies, to some degree, on patient accounts. However, the vast majority of patients with psychotic disorders deny having a mental illness^1,2^ from first presentation,^48^ suggesting that insight and psychopathology are semi-independent domains.^49^ Nonetheless, no technology, including artificial intelligence, enables clinicians to read the thoughts of patients, thus leaving insight assessment based solely on traditional face-to-face approaches such as interviews and questionnaires, which may increase the likelihood of a patient adapting their answers to satisfy the clinician or researcher. Responses may be indicative of conflicting motivational dynamics: individuals may deny an illness they recognise as a means of preserving self-image or, alternatively, express acceptance of an illness they do not genuinely endorse, potentially to satisfy perceived expectations from clinicians or to increase the likelihood of hospital discharge. Some degree of expertise from the interviewer is therefore needed. While self-report measures of insight may reduce any censorship, they may provide an overly simplistic measure of insight. Also, previous results for the agreement between researcher-rated and self-report measures are mixed.^50^ Of relevance, the present SUMD-based proxy measure for assessment of insight in psychosis may have overcome these issues. In particular, it is worth noting that no cooperation from patients – this validation study was based on EHR – or significant rater experience were required, while the proxy SUMD has been shown to correlate strongly with the three insight dimensions from SUMD, namely awareness of having a mental illness, awareness of the need for treatment and awareness of the social consequences of the disorder.

Methodological limitations

Three potential limitations should be considered when interpreting the study findings. First, this proxy instrument provided an unidimensional, three-point Likert score to measure such a complex phenomenological construct as insight in psychosis, which may therefore fail to capture its multidimensionality. It is important to note, however, that this unidimensional proxy measure of insight has been shown to have a robust correlation with the three principal insight domains of SUMD. Second, this study relied on previously collected data from the PAFIP programme, and only the three main SUMD items were considered, which may have affected content validity. More specifically, potential correlations of insight levels with severity of specific symptoms, such as hallucinatory experiences, delusions, disorganised thoughts, blunted affect, anhedonia or lack of sociability, could not be assessed. This is a common limitation of studies using SUMD, given the great heterogeneity in its administration.^41^ Third, the proxy tool was validated using EHR from a FEP cohort receiving care under an EIS, which tend to be more resourced than non-EIS mental health settings,^51^ including higher-quality documentation, which may have limited the generalisability of our results. Finally, inclusion of an external validation cohort was not feasible, which should be acknowledged as an important limitation of the present study.

Clinical implications and future directions for research

The validation of this proxy version for SUMD has clinical implications. Since the original SUMD has been primarily employed in clinical trials and epidemiological studies, and to a lesser extent in clinical practice, this proxy version may aid in generating insight data from real-world patients, thus making a major contribution to practice-oriented research. This proxy measure may also be useful in standardising clinical data from case report-based systematic reviews, thereby facilitating research into less common psychotic disorders such as shared delusional disorder or Huntington’s disease schizophrenia-like psychosis. Also, this EHR-based proxy SUMD may have implications for management – for instance, as an outcome measure when auditing and monitoring mental health services in the psychosis pathway. The observed correlations between the proxy and other validated psychometric instruments, such as BPRS, SAPS and SANS, replicated the well-known relationship between lack of insight and psychotic symptom severity – poorer insight and worse psychosis.^52^ However, future studies should investigate whether this proxy for insight can also predict other outcomes such as psychotic relapse,^53^ suicidal behaviour^15^ and/or quality of life.^18^ Whether the SUMD proxy can be administered by other mental health professionals, such as nurses or social workers, remains to be demonstrated. In addition, it would be interesting to replicate these findings in less resourced mental health services – for instance, in low- and middle-income countries. Finally, this proxy may be automated in EHR-based research applying natural language-processing techniques,^54^ which requires further research.

In conclusion, while the assessment of clinical insight is paramount in the management of patients with psychotic disorders, previous validated psychometric scales remain underutilised in daily practice, representing an unmet clinical need. Alternatively, the proposed SUMD-based proxy tool for assessment of insight using EHR notes may offer a feasible, valid and reliable alternative to standard psychometric procedures for measuring insight in real-world settings. This proxy SUMD may be a particularly valuable tool in patient-oriented research, thus bridging the gap between research and routine clinical care in mental health.