Psychometric Properties of the COVID-19 Stress Scale in College Students

TL;DR

This paper examines how well a stress scale works for college students during the pandemic, suggesting changes to improve its accuracy.

Contribution

The study evaluates and modifies the psychometric properties of the COVID-19 Stress Scale for better use in college students.

Findings

The original factor structure of the CSS was partially supported.

A five-factor structure is suggested with removal of outdated items.

The CSS can be improved for measuring stress in future health crises.

Abstract

Many experienced isolation and restricted behaviors due to the rapid onset of the COVID-19 pandemic in 2020. Investigations related to the psychological factors such as stress along with the danger of spread and contamination are scarce. The COVID-19 stress scale (CSS) was developed in order to evaluate such stress and anxiety related to the COVID-19 pandemic. The current study investigated the psychometric properties of the CSS, using a survey to provide evidence towards its continued use as a scientifically sound measuring instrument for future acute health crises in a sample of 615 college students (78.80% female, 18.60% male, 1.30% trans male, 1.00% non-binary), with a mean age of 19.10 years. The study partially supported the original measure’s factor structure. The main modification suggests a five-factor structure and removal of items related to less frequently used methods of banking and postal mail. The authors provide suggestions for future validation directions, use of the CSS and development of stress response strategies for students.

Full text

1. Introduction

In 2020, the United States, among many other countries across the world, experienced an unprecedented time of isolation and restricted behaviors due to a rapidly spreading COVID-19 pandemic. After virtual-only instruction during mid-March to July of 2020, many universities, including our mid-sized Midwestern university, returned to in-person or hybrid style classroom learning experiences and campus activities in Fall 2020. Pandemic levels were still high, causing concern among students, faculty, and staff. Debates ensued over the feasibility of keeping students safe and providing them with the education they paid for. Our university adopted several policies including reduced classroom size, social/physical distancing (Šuvaković, 2020), physical barriers, and paper mask dispensers. The university distributed two cloth masks to all students and employees. Mask usage was mandatory on all University properties. A close contact notification system was used to inform students if they needed to quarantine due to contact with the virus. Consequences for not following mask usage and other mandatory safety policies resulted in referral to the Office of the Dean of Students.

The Centers for Disease Control (2020) urged the population to make efforts to manage stress and protect their mental health during such tumultuous and strenuous times (National Center for Immunization and Respiratory Diseases, 2020). Accordingly, researchers began to investigate COVID-19 and its prevention measures’ impact on individuals’ mental well-being. Psychological stress as both a catalyst and the result of serious health concerns or illness is not new (Ovsiannikova et al., 2024; Salleh, 2008). Eubank and colleagues examined the impact of COVID-19 stress in students early on, using the Perceived Stress Scale (PSS-12; Eubank et al., 2021), and found it psychometrically valid and able to capture the high levels of anxiety and depression in Black, Indigenous, and People of Color (BIPOC) respondents compared to Caucasian respondents. Additional investigations into psychological factors associated with beliefs and behaviors related to spreading and containing viruses are needed to further understand and predict health-related behaviors (Taylor, 2022). The CSS (Taylor et al., 2020a) was developed in order to evaluate such stress and anxiety related to the COVID-19.

There are various perspectives to assess the role of stress and its risk of disease. Stress can be discussed biologically as the result of the functioning of specific physiological systems in the body that are particularly responsive to psychologically and physically demanding conditions. Repeated and continuous activation of such systems over time (i.e., sympathetic nervous systems) takes a toll on the body and increases the risk of illness (O’Connor et al., 2021; Yaribeygi et al., 2017). An environmental stress perspective, on the other hand, suggests that our assessments of our surroundings and environmental situations, along with subsequent experiences, can lead to substantial demands. The accumulation of demand from life events, particularly those related to health, change, and uncertainty, is related to several physical conditions (Arza et al., 2019; Mariotti, 2015). However, evaluations of events are subjective, and one’s ability to cope with demands can vary depending on the situation or experiences. This more psychological view of stress emphasizes that if a problem or situation is deemed serious and there is a lack of resources, stress ensues (Mattioli et al., 2020).

The experience of stress can be viewed through the transactional model (Lazarus & Folkman, 1984), where individual differences mediate the response to stress. How someone views COVID-19 influences the extent to which they experience stress. In this model the primary appraisal is the perceived threat, and the secondary appraisals are the coping mechanisms available to adequately manage that stress (Lazarus et al., 1985). Overall, the perceived stress caused by COVID-19 could vary across individuals, and the CSS could be a method of measuring these variations.

Regardless of the perspective subscribed to, COVID-19 can be viewed as a stressful life event if not a traumatic event. A recent study reported the presence of traumatic stress reactions in individuals directly as well as indirectly exposed to the virus (Bridgland et al., 2021). The pandemic spread rapidly, prompting health and government officials to scramble for resources to combat its spread. Uncertainties also loomed regarding COVID-19 severity, length, and treatment. Events that are beyond one’s control and seem endless are associated with greater stress and reduced immunity (Hussein Kahya & Mahmood, 2025). Indeed, they discuss a shift in immune system functioning from adaptive to potentially detrimental with continued stress exposure. Moreover, COVID-19 resulted in restricted measures such as lockdowns, social/physical distancing, and mask mandates. For the first time in a long time, the US population faced an unpredictable and rapidly progressing situation that confined or restricted their movement both within and outside the country, thereby impeding their ability to engage in typical social activities. Confinement and social isolation are linked to greater experiences of negative emotions such as fear, frustration, anger, and anxiety with potentially long-lasting effects on one’s psychological well-being (Brooks et al., 2020; Loades et al., 2020).

Fear of contracting COVID-19 or falling ill, of encountering infected objects or individuals, of dying, of suffering if infected (economically and socially), of loneliness and stigma, as well as feelings of helplessness are common psychological reactions noted in other viral outbreaks (i.e., Ebola, H1N1, MERS) that can intensify with restrictions and mandates enacted to help attenuate the rate of a viral spread (Al Najjar et al., 2016; Taylor, 2019; Van Bortel et al., 2016). Although empirical evidence links epidemics with stress, investigations into psychological factors associated with beliefs and behaviors related to spreading and containing viruses are scarce (Taylor, 2022). Such investigations are needed because pandemic-related stressors can exacerbate various types of preexisting mental health problems and increase the risk of anxiety and depression even in individuals with no prior history of mental health disorders (Asmundson et al., 2020; Rajiv Bhatla, 2021). Increased stress may also lead to maladaptive behaviors to cope with stress and anxiety, such as self-medicating (i.e., drugs) or self-harm (Goreis et al., 2023; Sinha, 2024). The steadily increasing pandemic research suggests that there will be significant long-term mental health needs (Bourmistrova et al., 2022). Data related to anxiety from previous epidemics, such as SARS and H1N1 (swine flu), indicated that anxiety was a key driver of behavior. Too much anxiety resulted in behaviors like over-buying household items, and too little anxiety resulted in less concern for public health measures (i.e., handwashing and vaccination uptake) (Asmundson & Taylor, 2020a, 2020b; Taylor, 2019; Taylor et al., 2020b). These data have helped predict health behaviors and can help plan public health messaging and intervention efforts (Taylor, 2019). Hence, reliable and valid measurement of COVID-19-related stress is important to further understand its relation to psychological and physical well-being.

In order to assess COVID-19-related stress, Taylor et al. (2020a) created the CSS, a 36-item measure consisting of sub-scales of danger and contamination fears, socio-economic consequences, xenophobia, compulsive checking and reassurance seeking, and traumatic stress symptoms. These subscales indicated the extent to which respondents experienced COVID-19-related worries as well as how often respondents engaged in compulsive checking behaviors, or how frequently they experienced problems related to traumatic stress in the past week. While other measures seek to capture fear and anxiety around COVID-19 (Ahorsu et al., 2020; McKay et al., 2020), the CSS captures many dimensions. The items were selected based on previous data (Taylor, 2019) and consultation with experts on health anxiety. Initial validation of the measure was conducted with population-representative adult samples from Canada and the United States (Taylor et al., 2020a). Overall, scales performed well on various indices of reliability and validity and offered promise as a tool to understand the types of distress associated with COVID-19 and to identify individuals at risk.

The current study sought to provide further evidence of the CSS psychometric properties in a younger population, college students. Taylor et al.’s (2020a) sample’s mean age was close to 50 years of age (M = 49.80; SD = 16.20), and about half of the participants were employed (52.30%). Only 4.4% of the sample were students. The American Institute of Stress (Tarsitano, 2019) has described stress among college students as an epidemic spreading across the country, with the American College Health Association (2020) reporting that three out of four college students self-report feeling stressed. Stress experienced by college students is multi-faceted with a variety of contributing factors, but academic-related stressors are thought to play a major role (Barbayannis et al., 2022; Beiter et al., 2015). Beginning and attending college represents a unique developmental stage in a young person’s life (Arnett, 2007; Schwartz, 2016). This period of emerging adulthood may be a time of increased well-being (Potterton et al., 2022), while others experience greater anxiety and depression (Tariq et al., 2021). Hence, a college student population is critical to investigate because they may have different concerns and anxieties compared to older and more occupationally diverse populations.

Student burnout, a condition resulting from chronic stress and overwhelming demands (Maslach & Jackson, 1981), saw an increase across countries as a result of COVID-19 stressors (Abraham et al., 2024). Burnout tends to vary as a function of gender, as girls have been shown to experience higher levels compared to boys (Romano et al., 2022). Similarly to the perception of stress, individuals may experience burnout differently. Overall, burnout has been linked to many negative impacts on health, academic performance, self-esteem, use of positive coping strategies (March-Amengual et al., 2022), and completion of academic program (Bask & Salmela-Aro, 2013).

College populations throughout the US may have interpreted stress differently because they were in a setting where mask wearing and other COVID-19 precautions were enforced. The current study examined the factor model of the CSS in a university population with these COVID-19 safety mandates. More research is needed to fully understand the psychological consequences of the pandemic in different populations and, accordingly, formulate psychometrically sound measures to help inform psychological intervention (Xiang et al., 2020). The current study investigates the psychometric properties of the CSS to provide evidence towards its continued use as a scientifically sound measuring instrument for future epidemics.

2. Materials and Methods

2.1. Sample Participants and Data Collection Procedures

An anonymous survey was administered using Qualtrics and distributed across the University to all students, faculty and staff through the Institutional Review Board (IRB) approved recruitment tools (paper flyers and digital campus communication center notifications). Students enrolled in the Psychological Science 100-level course were offered course credit if they completed the survey. No monetary or tangible incentives were offered for those not enrolled in a Psychological Science 100-level course. The survey and project were approved by the Ball State University IRB, protocol number 1668284.

2.2. Measures

The CSS is a 36-item measure grouped into five domains, including: (1) danger and contamination (12-items), (2) socio-economic consequences (6-items), (3) xenophobia (6-items), (4) traumatic stress symptoms (6-items), and (5) compulsive checking and reassurance seeking (6-items). Items are rated on a 5-point Likert scale ranging from 0 (not at all) to 4 (extremely) for fear-related items and rated from 0 (never) to 4 (almost always) for frequency-related questions. The scores for each of the five domains are calculated as the sum of ratings for each item in that domain. Table 1 shows the CSS descriptive information and reliability. Higher scores indicate more intense or more frequent perceptions. The English version of the CSS was used in this study with no adaptations or added/removed items.

2.3. Analysis Plan

To assess the psychometric properties of the CSS, confirmatory factor analysis (CFA) was performed on the original five-factor structure proposed by Taylor et al. (2020a). Data analysis was computed on complete cases (listwise deletion used for missing cases). Model fit statistics, factor loadings, and modification indices were requested. Cronbach’s alpha tests of internal consistency were also performed for each factor. Based on the fit and suggested modifications, subsequent factor structures were tested using exploratory factor analysis. All analyses were run using the R statistical software platform version 4.4.1 (R Core Team, 2021) and the Lavaan package version 0.6-19.

3. Results

3.1. Sample Characterization

A total of 650 participants took part in the survey, which was available from November 2020 to March 2022. This timeframe was chosen because university precautions for masking, distancing, and changes in class size were in effect. From that total, 35 were removed due to incomplete survey responses, leaving 615 responses in the final data set. The mean age of respondents was 19.10 years, and a majority identified as female (78.80%). The remaining respondents identified as male (18.60%), trans-male (1.30%), and non-binary (1.00%).

3.2. Data Analysis

Cronbach’s alpha for the six original subscales of the CSS indicates good internal consistency (danger and contamination = 0.89, contamination = 0.89, socio-economic consequences = 0.92, xenophobia = 0.95, traumatic stress symptoms = 0.89, compulsive checking = 0.83). Results of CFA on the original six-factor structure of the CSS appear in Table 2.

To evaluate model fit, the following criteria for good fit were used: Chi-Square p > 0.05. CFI/TLI > 0.90, RMSEA < 0.05.

Model fit statistics indicate poor fit (Chi-Square = 1337.70, p < 0.01, CFI = 0.83, TLI = 0.82, RMSEA = 0.09). To investigate the poor fit and suggest an alternative structure, modification indices for the original CFA as well as an Exploratory Factor Analysis (EFA) were run on the entire battery of items. The results of the EFA appear in Table 3.

Results of the EFA, in general, supported the structure of the original socio-economic consequences, xenophobia, traumatic stress symptoms, and compulsive checking. Examination of the loadings indicates that the items from the original Contamination factor had generally low to moderate loadings for all items. Items 19, 20, and 21 had only moderate loadings on the original Contamination factor but loaded reasonably well with the items on the original Danger factor (γ = 0.67, γ = 0.42, γ = 0.56). These items assess worry about catching the virus from other people or from things in public spaces.

Modification indices from the original CFA generally agree with the results of the current CFA. Like Taylor et al. (2020a), evidence from both the EFA and the modification indices indicated that the contamination factor could be combined with the danger factor to create a five-factor solution. To get the best fit, however, the modification indices also indicated that certain error terms should be correlated that were not noted in the original model (see Table 4). Furthermore, items 22, 23, and 24 did not load onto the danger and contamination factor as previously reported in Taylor et al. (2020a). Items 22, 23, and 24, which ask about physically handling currency and mail, loaded well together; however, three items can be considered a bare minimum for the definition of a psychometrically sound factor (Kline, 2000). Given the items only loaded well with each other (but to no other factors) and the somewhat antiquated nature of these issues in current culture, the decision was made to delete the items instead of retaining a sixth factor.

3.3. Modified Five Factor Structure

The final modified structure contains five factors. Cronbach alphas for the modified five subscales all indicate good internal consistency: danger and contamination (γ = 0.92), socio-economic consequences (γ = 0.92), xenophobia (γ = 0.95), traumatic stress symptoms (γ = 0.89) and compulsive checking (γ = 0.83). Based on the results of the EFA and modification indices, items 22, 23 and 24 were removed entirely from the model. In addition to the item changes, modification indices recommended correlating errors between multiple pairs of variables (See Table 5). CFA results appear in Table 3. Fit for the modified five factor structure was good (Chi-Square = 801.90, p < 0.01, CFI = 0.92, TLI = 0.91, RMSEA = 0.07).

4. Discussion

The proposed factor structure of the original 36-item CSS consisted of a five-factor solution that included the following: (1) danger and contamination fears, (2) fears about economic consequences, (3) xenophobia, (4) compulsive checking and reassurance seeking, and (5) traumatic stress symptoms about COVID-19. The average age of the survey respondents was almost 50 years old, and the majority had completed a full or partial college degree. The creators of the CSS (Taylor et al., 2020a) have called for additional research to expand the breadth of content of the measure. The aim of the current study was to evaluated the CSS’s fit in a college student sample returning to in-person instruction in the Fall of 2020. All students, faculty, and staff were required to use masks while in University or University-associated settings and to maintain physical distancing (Šuvaković, 2020) in classes and University-associated gatherings. Consequences were also associated with not abiding by these university COVID-19 policies.

Although not many, published psychometric studies of the CSS report mixed evidence, either suggesting a 5-item or 6-item solution depending on the population (i.e., Adamczyk et al., 2021; Milic et al., 2021). Results partially supported the original five-factor model. The authors recommend considering reducing the 36-item survey to 33 items and slightly modifying one of the scales. Items assessing one’s worry and fears still fit well under pandemic danger and contamination. However, the CFA fit and loading indices indicated the need to reorganize and remove several items from this scale for an acceptable model fit. Items assessing the risk of catching the virus from handling money or mail and engaging in cash transactions may be less prevalent now than in the past; however, this may be a function of age and country. Even before COVID-19, mobile payments and debit cards surpassed cash as the most prevalent payment method used by US college students (Sallie & Ipsos, 2019). Although the proportion of Americans whose typical week includes mostly cash purchases has steadily decreased from 18% in 2018 to 14% in 2022 (Faverio, 2022), this trend of cashless payments may not hold up in countries other than the US, as cash usage is variable internationally (Ashworth & Goodhart, 2020). A study examining a cross-section of college students reported that postal mail was one of the least preferred methods of communication, with email and texting being most preferred for academic and non-academic communications (Swanson et al., 2020). Individuals 35 years of age and under also send and receive fewer pieces of postal mail and tend to adopt newer and faster communication technologies (United States Postal Service, 2018); however, postal service usage may vary across age groups and countries (Khazabi, 2017).

This study is one of the first large-scale validations of mainly college students in the United States (US). It also provides support for the psychometric structure of the CSS. Some limitations of this study include reliance on a convenience sample, absence of test–retest reliability and lack of measurement invariance testing. Further studies are also required to support the construct validity of the CSS, particularly as it relates to post COVID-19 life and reliance on digital communication. The CSS, nevertheless, offers promise as a tool for better understanding the distress associated with pandemics and for identifying people in need of mental health services. Given that approximately 80% of college students report feeling stressed sometimes or often, and over 70% of American college students report a decline in mental health during the pandemic (Tarsitano, 2019; Son et al., 2020), psychometrically sound measures are vital for continued research and assessment improvement. Psychological issues can have a major impact on college students, including drop-out, poor health outcomes and lower job placement (Hjorth et al., 2016).

It is important to note that the study’s sample primarily consisted of freshmen transitioning into a Midwestern college with COVID-19 imposed restrictions on their learning and socialization. A shortened 33 item CSS may allow researchers to better capture the most critical issues related to each factor for different populations, and under different conditions. The data in the present study were gathered while mandates like mask wearing were in place on campus. Health mandates in response to COVID-19 varied across countries (Yan et al., 2020) and across states in the US (Zhang & Warner, 2020), and these factors may have an impact on the dimensions captured by the CSS.

It should also be noted that the sample used in this study was very imbalanced in terms of gender. In addition to being an issue of sample generalizability, this imbalance made checking measurement invariance across genders not possible due to the small number of males included in the study. A sample with greater balance across genders would be beneficial because previous studies have shown that stress appraisal varies, with women reporting higher levels of stress compared to men (Barbosa-Leiker et al., 2013). It is possible that gender-based differences in stress appraisal had an impact on the current data set, considering 78.80% of the respondents were female.

Future investigations on the CSS’s psychometrics are needed to better understand its item and factor structure in different contexts, as all college campuses are not homogeneous. For example, a survey of college students attending a university in the Bronx indicated female-identifying BIPOC students showed higher resilience compared to students of other genders/races (Burt et al., 2022). A previous study compared the experiences during the pandemic in college student in two countries, Argentina and the US (Rice et al., 2023) and found differences in COVID-related fear and stress between countries. The CSS could also be used to explore differences across counties and age groups and the implications for social support and coping strategies.

5. Conclusions

The current study examined the psychometric properties of the CSS in a US college sample. The CSS continues to show promise as a stress-measuring instrument for future epidemics and would benefit from additional cross-cultural and longitudinal validation. Future evaluations of the CSS and its findings may provide valuable stress measures and be critical in developing and providing student mental health support services and activities that support mental health, reduce stress and address student burnout (Burt & Eubank, 2021; Eubank et al., 2024). Although the current study suggests the removal of CSS items dealing with cash transactions and use of postal mail, additional research is needed to examine its breadth of content and validity across other age groups and countries.