Development of a questionnaire for problematic social networking sites use: Ensuring content validity through Delphi methodology

TL;DR

This paper describes the development of a validated questionnaire to assess problematic social media use in both adolescents and adults.

Contribution

A new multidimensional PSNSU questionnaire was developed using Delphi methodology to ensure content validity across age groups.

Findings

The PSNSU questionnaire was refined through three Delphi rounds, resulting in two domains, 10 subdomains, and 42 items.

The final questionnaire achieved high content validity with an average CVR of 0.99 and consensus of 0.97.

The tool captures addictive behaviors and occupational challenges related to social media use across life stages.

Abstract

This study aimed to develop a questionnaire to assess problematic social network service use (PSNSU) applicable to both adolescents and adults to ensure content validity.

A Delphi survey methodology with a panel of 16 experts was employed from April to June 2024 to ensure the content validity of the PSNSU assessment tool. This study involved three rounds of Delphi surveys to collect both open- and closed-ended responses to the PSNSU questionnaire. Data analysis focused on calculating the content validity ratio (CVR), stability, and consensus of each item. Items that did not meet the threshold criteria were revised or removed based on expert feedback.

The PSNSU questionnaire initially included three domains and 59 items. Following Round 1 of the Delphi survey, three domains, 11 subdomains, and 43 items were selected. After adding and modifying the domains and items, Round 2 resulted in a questionnaire with two domains, 10 subdomains, and 42 items. In Round 3, the revised PSNSU questionnaire showed an average CVR of 0.99, a stability of 0.10, and consensus of 0.97, thus concluding the Delphi process.

The final PSNSU questionnaire encompasses a wide range of issues related to addictive behaviors and occupational challenges. Content validity was validated through the Delphi survey, resulting in a comprehensive tool that reflects the multidimensional characteristics of PSNSU across different life stages. This tool can be used for both clinical assessments and research to better understand and address PSNSU.

Figures

Full text

Introduction

Social Network Services (SNS) have experienced rapid growth with the advancement of the Internet [1] and have become major communication tools because they allow users to communicate and interact with each other [2]. SNS have become globally and deeply embedded in individuals’ daily lives, with over five billion people using social media worldwide in 2024, and this number is projected to increase to almost six billion by 2028 [3]. People are motivated to use SNS for social interaction, information seeking, passing time, entertainment, relaxation, and information sharing [4]. However, excessive or problematic use of SNS (PSNSU) has been identified as a significant concern, leading to adverse effects and serious social issues [5].

PSNSU is characterized by addictive behaviors such as mood modification, salience, tolerance, withdrawal, and relapse/conflict [6], which impair various aspects of users’ lives, including educational and work performance, personal relationships, and social interactions [6,7]. Additionally, it can reduce offline social networking experiences, contribute to social comparison and feelings of inferiority [8], and cause issues such as peer phubbing [9]. Given these impacts, adolescents and adults are particularly vulnerable to PSNSU issues [10,11]. Adolescents are particularly susceptible to PSNSU because of the developmental stage of their brains, which may impair impulse regulation [12,13]. Similarly, due to COVID-19, young adults have developed a lifestyle characterized by the excessive use of SNS in daily living [11,14]. As SNS permeate various aspects of everyday life for both adolescents and young adults [7,15], assessing not only the addictive aspect but also the broader negative impact on their lives is crucial for understanding PSNSU.

However, existing assessments of PSNSU often address the issue from a single dimension and fail to fully reflect its complex and multifaceted nature. For instance, some studies, such as those utilizing the well-known Bergen Social Media Addiction Scale [16], focus solely on addictive aspects by assessing six items in the one-dimensional concepts of PSNSU. The Social Media Disorder Scale [17], also primarily measures addictive behaviors without considering the broader life impacts. While the addictive aspect has been sufficiently considered in previous studies, occupational aspects, understood as those related to daily living activities such as worsening educational performance, compromised work performance, difficulties in managing physical and psychological health, and poor rest and sleep quality [7,18], along with social aspects such as relationships with others and participation in social activities [7,11], are often overlooked in assessments.

To comprehensively understand PSNSU and its impact on users’ real lives, it is crucial to organize the PSNSU assessment to include not only the addiction domain, but also occupational and social problem domains. Therefore, it is essential to incorporate a wider range of impact areas into the PSNSU assessment [19]. Specifically, the occupational problems domain should cover issues such as education, work, health management, rest, and sleep problems [20], whereas the social problems domain should include aspects such as social comparison and peer phubbing [21]. This approach highlights the need for holistic and comprehensive tools that can capture the broad spectrum of problems caused by PSNSU, encompassing both occupational and social issues.

Moreover, existing scales and questionnaires are often limited to specific age groups such as adolescents or university students, leading to a lack of comprehensive tools that address both adolescents and adults. For example, the Problematic Use of SNS Scale [22] was developed for adolescents aged 15 to 18 years, the SNS Use Disorder Test-10 [23] for university students, and the Smartphone Addiction Scale [24] and SNS Addiction Proneness Scale [25] for college students. Consequently, these tools do not provide a holistic view of PSNSU across different life stages. This division creates challenges for comprehensively understanding the state of PSNSU and developing tailored intervention strategies. Consequently, a need arises for an assessment tool that includes both adolescents and adults and accurately reflects the multidimensional characteristics of PSNSU.

This study aimed to develop the PSNSU questionnaire to assess three main domains: addiction, occupational problems, and social problems. This tool was designed for use with both adolescents and adults using the Delphi survey methodology. Specifically, this study sought to ensure the content validity of the PSNSU assessment tool by incorporating the opinions of an expert panel.

Methods

Study design

A Delphi survey was conducted to ensure the content validity [26] of the PSNSU assessment tool by collecting both open- and closed-ended responses to the PSNSU questionnaire from a panel of experts. The Delphi method is a structured communication technique originally developed as a systematic, interactive forecasting method that relies on a panel of experts [27]. It is commonly used to collect and distill the knowledge and opinions of experts through a series of questionnaires interspersed with controlled opinion feedback [28,29].

PSNSU questionnaire

The preliminary PSNSU questionnaire, developed for adolescents and young adults in the Republic of Korea, was created before achieving a content validity consensus through the Delphi methodology. The questionnaire includes three domains, 11 subdomains, and 59 items. The addiction domain consists of six subdomains: mood modification (seven items), salience (five items), tolerance (seven items), withdrawal (six items), and relapse (three items). The occupational problems domain comprises three subdomains: education/work problems (five items), health management issues (six items), and rest and sleep problems (five items). The social problem domain includes three subdomains: SNS-centered social networks (three items), social comparison (five items), and peer phubbing (seven items). This structure comprehensively assesses the multifaceted aspects of PSNSU, including addiction, occupational challenges, and social problems.

Participants

The criteria for the expert panel in this study included: 1) experts with specialized knowledge in the field of digital addiction and mental health for adolescents and adults; 2) individuals with at least five years of clinical or research experience in the relevant field (e.g., professors, psychiatrists, occupational therapists, and researchers); 3) those who could use email for approximately two months during the Delphi period; 4) individuals with a high understanding of the Delphi survey and who could actively and continuously participate in the research; and 5) those who voluntarily agreed to participate in the study. The expert panel was recruited from 06/04/2024 to 13/04/2024. Recruitment and consent for participation were obtained via email. Sixteen individuals were recruited, and provided their written informed consent. This study was approved by the Institutional Review Board of Yonsei University Mirae Campus (approval number: 1041849-202403-SB-047-01).

Data collection

Data were collected from 04/15/2024 to 06/05/2024, exclusively from experts who consented to participate. Before starting the Delphi survey, participants’ general characteristics (age, sex, education level, current occupation, and career duration) were collected. After collecting the general information, the Delphi survey was conducted in three rounds (Table 1). In each round, responses were collected on the relevance of the domain, subdomain, and items of the preliminary PSNSU questionnaire using a 4-point Likert scale (1, very irrelevant; 4 very relevant), along with open-ended feedback on the points to be revised (S1 file). Each round took 2–3 weeks to complete.

Table 1: Data Collection and Research Process.

Round 1.

The first round of the Delphi survey was conducted to assess the appropriateness of PSNSU. The domain section of the Round-1 survey consisted of 11 closed-ended questions and one open-ended question on the appropriateness of the domains. Closed-ended questions were measured on a 4-point Likert scale (Table 2), while open-ended questions gathered additional feedback on the operational definitions and conceptual distinctions of the domains and subdomains. The item section included 59 closed-ended questions and one open-ended question, measured in a similar manner.

Table 2: Closed-ended Questions of the Addiction Domains in PSNSU Questionnaire of the Delphi Round 1.

Round 2.

Based on feedback from Round 1, the operational definitions of some subdomains were clarified, and items were removed or revised for Round 2 of the Delphi survey. The domain section of the Round 2 survey included 11 closed-ended questions and one open-ended question in the domain and subdomains. The item section included 43 closed-ended questions and one open-ended question, measured in the same manner as in Round 1.

Round 3.

Round 3 of the Delphi survey was conducted by restructuring the domains, subdomains, and items based on the results of Round 2. The first section of the Round 3 survey included 10 closed-ended questions and one open-ended question on the domains and subdomains. The second section included 42 closed-ended questions and one open-ended question, measured in a manner similar to that in Round 1.

Statistical analysis

The general characteristics of the experts were analyzed using descriptive statistics with SAS software. Results from all rounds of the Delphi survey were analyzed using means, standard deviations, content validity ratios (CVR), stability, and consensus based on the closed-ended response data. Means and standard deviations were calculated from the responses collected from a 4-point Likert scale. CVR is defined as ( $[eqn]$ )/( $[eqn]$ ), where $[eqn]$ is the number of experts rating the item as essential, and N is the total number of experts [30]. It is appropriate when above 0.49 with a panel of 16 experts [30]. Items below the CVR threshold were removed in this study. Stability, indicating experts’ agreement on each item, was analyzed using the coefficient of variation [31]. A coefficient less than 0.5 indicates good stability [31]. Consensus is defined as the level of agreement among experts, typically measured by calculating the percentage of agreement or using statistical measures such as the interquartile range (IQR) [32]. A threshold of consensus is often set at 75% agreement [33]. In this study, items not meeting the stability and consensus thresholds were revised or removed based on open-ended comments. The Delphi method concluded when all items met the thresholds of a CVR above 0.49, stability below 0.5, and consensus above 0.75, ensuring content validity. Excel was used for analyzing the Delphi data.

Results

General characteristics

The general characteristics of the 16 experts included in this study are presented in Table 3. The average age of the experts was 37.81 years, with nine females (56.25%). The majority of participants held a Ph.D. degree (11 participants, 68.75%) and were employed as professors (six participants, 37.50%). The average duration of research and clinical experience in addiction and mental health was 12.19 years.

Table 3: General Characteristics of the Delphi Experts.

Round 1 results.

Round 1 of the Delphi survey was conducted (Table 4). In the domain section of the Delphi survey, 11 domains showed an average CVR of 0.91, stability of 0.14, and consensus of 0.90. Among the three indicators, the consensus for the SNS-centered social network (0.71) and peer phubbing (0.71) were below the threshold and were removed. Open comments on the low consensus suggest the need to further specify the meanings of both subdomains. Consequently, the term “SNS-centered social network” was clarified to mean “an individual’s preference toward using SNS for maintaining and establishing relationships,” rather than “a social network centered around SNS.” Similarly, “peer phubbing” was clarified to mean “focusing on SNS over interactions with others in social situations,” rather than “ignoring interactions with others in social settings due to SNS usage.”

Table 4: Results of Delphi Round 1.

In the item section, 59 items showed an average CVR of 0.83, stability of 0.18, and consensus of 0.78. Among the CVR values, health management issue items 1 (0.25) and 3 (0.38), as well as rest and sleep problems item 4 (0.38), were below the threshold and were deleted. In total, 27 items were found to be below the consensus threshold. Open comments on the low consensus indicated that 17 items (mood modification items 4, 6, and 7; salience item 5; tolerance items 1 and 6; withdrawal items 3 and 4; social comparison items 3 and 5; peer phubbing items 1, 4, and 6; education/work problems item 3; health management issues items 1 and 3; and rest and sleep problems item 4 had overlapping meanings with other items within the same domain, suggesting their deletion. For the remaining 10 items (mood modification item 5; tolerance item 7; relapse items 2 and 3; health management issues items 4, 5, and 6; and rest and sleep problems items 3 and 5), it was recommended that their meanings be clarified. Additionally, it was suggested to add an item related to rest within the rest and sleep subdomains. Consequently, 17 items with overlapping meanings were deleted, 10 items had their meanings clarified, and a new item, rest and sleep problems item 5 “I cannot rest my body and mind properly due to SNS,” was added.

Round 2 results.

Round 2 of the Delphi survey was conducted based on the revised PSNSU questionnaire from Round 1 (Table 5). In the domain section, 11 domains showed an average CVR of 0.99, stability of 0.12, and consensus of 0.88. Among the three indicators, the consensus for SNS-centered social networks (0.71) and peer phubbing (0.71) was below the threshold, similar to that in Round 1. The open-ended comments suggested that there was no need to separate the two subdomains. Instead, it was recommended to integrate them into the social relationship and social participation problem subdomains within the occupational problem domain. Based on this feedback, a new social relationships problem subdomain was created in the occupational problem domain, operationally defined as “issues arising from the use of SNS that negatively impact the ability to maintain and establish personal relationships.” Additionally, a new participation in social activities problem subdomain was established, operationally defined as “challenges related to engaging in social activities due to prioritizing SNS use over face-to-face interactions.”

Table 5: Results of Delphi Round 2.

In the item section, the analysis of the 43 items (originally 59 items from Round 1, with 17 items removed and one item added) showed an average CVR of 0.94, stability of 0.19, and consensus of 0.88. The consensus for the following three items was below the threshold. First, it is recommended that tolerance item 5 be moved from the tolerance subdomain to the salience subdomain. Second, it was suggested to delete health management issue item 2, as it was difficult to interpret the item solely as a problem arising from PSNSU. Third, social comparison item 2 was recommended for deletion because of concerns regarding its appropriateness. Accordingly, these three items were either removed or deleted.

Additionally, items related to SNS-centered social networks and peer phubbing, which showed low consensus values up to Delphi Round 1, and all social comparison items, were below the threshold for consensus. The open-ended comments suggested deleting the social problem domain and moving all items within it to the occupational problem domain. Based on this feedback, all nine items (excluding the deleted social comparison item 2 due to its low consensus) included in the social problem domain were relocated. Five items (SNS-centered social network items 2 and 3 and peer phubbing items 1, 2, and 4) were moved to the social relationship problem subdomain. Two items (SNS-centered social network item 1 and peer phubbing item 3) were reassigned to the social participation problem subdomain. The remaining two items (social comparison items 1 and 3) were interpreted as having a psychosocial health management aspect and were moved to the health management problem subdomain.

Finally, the comments suggested the addition of items to measure the time and frequency of direct participation in activities and meetings as elements of social participation. Additionally, typographical and grammatical adjustments were made to improve the overall clarity based on feedback.

Round 3 results.

Delphi Round 3 was conducted based on the revised PSNSU questionnaire from Round 2 (Table 6). The analysis of the revised two domains and 10 subdomains showed an average CVR of 1.00, stability of 0.10, and consensus of 0.98, all meeting the threshold criteria. The analysis of the 42 items (43 items from Round 2, two items removed, and one item added) showed an average CVR of 0.99, a stability of 0.10, and a consensus of 0.97, all meeting the threshold criteria. Consequently, all domains and items satisfied the conditions of a CVR of 0.49 or higher, stability of 0.5 or lower, and consensus of 0.75 or higher, concluding the Delphi process. The CVR, stability, and consensus for all the Delphi rounds are listed in Table 7. The final questionnaire comprised 2 domains, 10 subdomains, and 42 items (S1 Table).

Table 6: The Results of Delphi Round 3.

Table 7: CVR, Stability, and Consensus Results for All Delphi Rounds.

Discussion

This study developed a questionnaire for the PSNSU targeting adolescents and young adults and verified its content validity using the Delphi survey methodology. In the final round of the Delphi survey, the PSNSU questionnaire demonstrated high content validity, with an average CVR of 1.00, stability of 0.10, consensus of 0.98, CVR of 0.99, stability of 0.10, and consensus of 0.97, indicating strong expert agreement on the relevance of the questionnaire content.

The multidomain structure of the PSNSU questionnaire validated in this study allows for a more holistic understanding of PSNSU. Previous research has primarily concentrated on addictive aspects as a single domain [16,17], neglecting the broader effects on everyday life. Addictive elements, such as mood modification, salience, tolerance, withdrawal, and relapse, which were not deleted by Delphi, have been extensively discussed in the context of substance addiction [34]. However, assessing digital addiction in modern society using only these addictive elements is limited [35,36] because it overlooks the multifaceted nature of PSNSU, which extends beyond addiction and impacts daily functioning and social interactions. Previous studies [19,37] reported that it is more appropriate to approach this issue from the perspective of problematic use rather than within an addiction framework to avoid misdiagnosing problematic use as addiction. This study fills this gap by developing a PSNSU assessment tool that not only evaluates addictive behaviors but also addresses associated everyday problems such as occupational difficulties and social issues, thereby providing a comprehensive measure of the impact of PSNSU on an individual’s life.

Through the Delphi process, the key domains were defined as addictive and occupational problems. Initially, the questionnaire included a social problem domain; however, it was consolidated into an occupational problem domain after obtaining expert consensus. Specifically, the SNS-centered social network, peer phubbing subdomains, and items from the social problem domain were moved to the social relationships and social participation problem subdomains within the occupational problem domain, providing a more cohesive structure. This adjustment demonstrated a high level of expert consensus and was aligned with the existing framework, which broadly describes occupational issues as those related to daily living activities [20].

Among the two domains, the occupational problem domain is particularly noteworthy, as it has been largely overlooked in the existing PSNSU scales and assessments. By including the occupational problem domain, SNS should be regarded as part of the daily lifestyle [11], as the tools can better capture the influences that are deeply related to the user’s everyday life. Subdomains such as education/work problems have been included in previous assessment tools [17,38], but they are often measured with a single item, limiting the ability to form a comprehensive factor. In this study, each subdomain is composed of at least three items, which is the recommended number of items per domain in a questionnaire [39,40]. This allowed for the evaluation of the comprehensive aspects of the PSNSU and facilitated the application of more systematic evaluations related to the user’s everyday life impact.

This study has several advantages. First, we developed a multidimensional tool for assessing PSNSU, providing a comprehensive understanding beyond the single-dimensional approach. This tool is significant because it evaluates not only the addictive aspects of PSNSU but also its impact on occupational problems in everyday life, which were previously unassessed by other tools. Second, the questionnaire targets both adolescents and adults, making it useful for measuring, tracking, and analyzing developing adolescents and adults using a single tool. This allows for age-specific assessments and provides data for tailoring intervention strategies. Third, the content validity of the questionnaire has been proven by a wide range of experts, making it applicable for systematically evaluating and managing PSNSU in various settings such as schools, workplaces, and communities, in academic research and practical environments.

This study has several limitations. First, it did not include a pilot testing phase with actual users, which was crucial for validating the practical applicability and reliability of the questionnaire. Previous studies [41,42] indicated that online connections can be healthier and more positive for users whose sexual preferences or gender identity is not accepted by their community. Therefore, future studies should ensure the validity of the PSNSU by clarifying the target study group and collecting data from a representative user population for comprehensive analysis. Second, the questionnaire developed in this study relies mainly on self-reporting, which may lead to a response bias based on the respondents’ subjective perceptions. To complement this, future research should include objective data (e.g., SNS usage time and usage patterns). Additionally, because of the nature of SNS, problematic behaviors may vary by platform [43,44]. However, this scale evaluates only broad SNS use, which may be insufficient for obtaining information about specific platforms or behaviors. Future research should identify response differences in the PSNSU questionnaire based on the SNS platforms used by the respondents. Third, although the content validity of this evaluation tool was verified through a Delphi study, the item structure of the PSNSU evaluation tool was not statistically confirmed. Therefore, confirmatory factor analysis and Receiver Operating Characteristic analysis are needed using data from a representative sample to verify whether the PSNSU questionnaire structures two domains—–10 subdomains and 42 items—–and has an appropriate response scale. In addition, the reliability and validity of the evaluation tool must be verified.

Conclusion

The final PSNSU questionnaire for adolescents and adults includes two domains and ten subdomains, encompassing a wide range of issues related to addictive behaviors and occupational challenges. The development and analysis of the PSNSU questionnaire using the Delphi survey methodology validated the content relevance of the two aspects of the PSNSU. By addressing the various dimensions of problematic SNS use and incorporating expert feedback, this questionnaire provides a valuable resource for both research and clinical practice, aiding in the identification and management of PSNSU across different life stages.

Supporting information

S1 FileRaw Data Collected in All Delphi Rounds.(XLSX)

S1 TableProblematic Social network Sites Usage Questionnaire with Confirmed Content Validity.(DOCX)