Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2024, Korea: a psychometric study

Abstract

Full text

The Korea Health Personnel Licensing Examination Institute administers national licensing examinations annually to determine whether candidates meet the essential competencies required for healthcare practice. To ensure that these examinations remain valid and appropriate, post-exam analyses of item difficulty, discrimination, and reliability are routinely conducted. Questions with extremely high or low difficulty or discrimination scores may require review and revision, offering meaningful insights for improving future tests. This study evaluated the item difficulty, discrimination, and reliability of 26 health personnel licensing exams held in Korea in 2024 to assess their overall suitability. It also explored the relationship between the average item difficulty and average item discrimination indices. This work follows a previous report covering the 2022 and 2023 licensing exams [1,2].

Since this study involved analyzing test results rather than human subjects, institutional review board approval and informed consent were not required.

Item analysis was performed on the 26 examinations using classical test theory. Item difficulty (P) was defined as the proportion of examinees who answered an item correctly, ranging from 0 to 1. Item discrimination was measured using the upper and lower 27% method, which compares the difference in difficulty between the top and bottom 27% of examinees, as well as the correlation of individual item scores with total test scores. Discrimination indices and reliability were calculated only for exams with more than 100 participants. Statistical analyses were conducted using IBM SPSS ver. 21.0 (IBM Corp.).

Pass rates across the 26 examinations varied widely, ranging from 25.0% to 100%. The number of examinees per exam ranged from 12 (for health educators–level 1 and midwives) to 181,890 (for care workers). Exams for physicians, dentists, midwives, nurses, doctors of Korean medicine, and pharmacists all had pass rates exceeding 90%. Notably, the midwives’ exam had a 100% pass rate, although it included only 12 candidates (Table 1).

Table 2 summarizes the results of the item analysis for the 2024 examinations. All exams showed high reliability, with the lowest reliability observed for the emergency medical technicians–level 2 exam (0.880). The average difficulty indices ranged from 53.1% (health educators–level 2) to 80.1% (physicians), while the average item-total correlations ranged from 0.19 (doctors of Korean medicine) to 0.43 (opticians).

According to classical test theory, average difficulty indices are classified as moderate (50%–60%), somewhat easy (60%–70%), easy (70%–80%), and very easy (80% or above). Based on these categories, the 2024 exams’ difficulty indices were rated as very easy for the occupational therapists and physicians exams, easy for 14 exams, and somewhat easy for 13 exams. For medical professions such as physicians (80.1%), dentists (77.3%), nurses (79.5%), and doctors of Korean medicine (74.6%), the difficulty indices fell into the very easy or easy categories, with corresponding pass rates exceeding 90%.

In classical test theory, discrimination indices below 0.30 are generally considered low. Seventeen examinations demonstrated low discrimination, whereas 11 showed good discrimination. For example, the doctors of Korean medicine exam had a relatively low discrimination index (average item-total correlation of 0.19), while the opticians exam demonstrated good discrimination (average item-total correlation of 0.41). All tests maintained high reliability, with Cronbach’s alpha values of 0.880 or higher, ensuring consistent measurement across examinations.

In 2022, the national licensing examinations had 468,352 candidates with an 89.4% pass rate. In 2023, there were 446,054 candidates with an 87.0% pass rate. In 2024, both the number of candidates (286,922) and the number of successful examinees (244,205) decreased, resulting in an 85.1% pass rate. The pass rate range shifted from 52.2%–100.0% in 2023 to 25.0%–100.0% in 2024. Analysis of difficulty indices for both years showed that most exams remained relatively easy. Twenty examinations in 2024 showed decreased difficulty compared to 2023, mainly shifting among the easy, very easy, and somewhat easy categories, without major overall changes. Additionally, the Care Workers Qualification Examination transitioned to computer-based testing in 2023, leading to non-disclosure of test items and analysis results, which limited direct comparisons.

In 2024, 17 examinations had lower discrimination indices than in 2023, and 12 showed decreases in item-total correlation coefficients. Despite these declines, the overall discrimination indices for 2024 did not differ significantly from those in 2023, suggesting no substantial reduction in discriminative capacity. Confidentiality policies restrict the release of detailed item analyses and test results, limiting comparisons with previous years and constraining broader academic discussion of these examinations.

In summary, Korea’s 2024 national health personnel licensing examinations exhibited satisfactory levels of difficulty, discrimination, and reliability. Nonetheless, 5 of the 26 exams had low discrimination indices despite having appropriate difficulty levels.