Detection of flawed multiple-choice questions in preclinical medical education using item difficulty and discrimination indices: a six-year analysis

TL;DR

This study shows that using statistical thresholds alone misses some flawed multiple-choice questions in medical exams, highlighting the need for expert review alongside quantitative analysis.

Contribution

The study provides empirical evidence that static psychometric thresholds miss a significant portion of flawed exam items.

Findings

14.3% of flawed items were missed when relying solely on p-value and rpb-value thresholds.

Flawed items tended to be more difficult and less discriminative than uncorrected items.

Expert review is necessary alongside quantitative metrics to ensure exam quality.

Abstract

MCQ exams may include flawed items affecting validity. Psychometric indicators such as item difficulty (p-value) and point-biserial coefficient (rpb-value) are widely used to identify problematic questions. Evidence on using p-value (< 0.25) and/or rpb-value thresholds (< 0) to detect flawed items remains limited. This study aimed to provide a proof-of-concept using a large, real-world dataset, evaluating how often flawed items were missed when relying solely on static thresholds.

Exam analyses from 32 preclinical courses (academic years 2017–2022) were reviewed. Items meeting predefined thresholds were flagged, while all items were manually reviewed when the most frequently chosen answer was not the keyed correct answer or when multiple options had similar p-values. Flagged items were sent to course directors for verification, and only confirmed items were recorded as corrections.

Among 236 exams, 59 (25.0%) required corrections, with at least 1 corrected item. Of 14,238 total items, 81 (0.6%) required correction, most due to ‘multiple-answers’ (46.9%), followed by ‘wrong-answer’ (40.7%), awarding points to all choices (‘all-choices’) (7.4%), and ‘item-removal’ (4.9%) causes. Of 77 corrected items with available p- and rpb-values (excluding ‘item-removal’), 66 (85.7%) met thresholds, while 11 (14.3%) had p-value ≥ 0.25 and rpb-value ≥ 0, indicating thresholds missed 14.3%. Corrected items had significantly lower p- and rpb-values than uncorrected items (P < 0.001). Item correction status showed negative correlations with both p- and rpb-values, suggesting flawed items tended to be more difficult and less discriminative. For the ‘wrong-answer’ and ‘multiple-answers’ causes, the actual correct answers had higher p-values than the initially designated ones (P < 0.001). For the ‘wrong-answer’ cause, the actual correct answers had higher rpb-values (P < 0.001), while they were comparable in the ‘multiple-answers’ cause, highlighting flaws even with unremarkable rpb-values.

While p- and/or rpb-value thresholds detected 85.7% of flawed items, 14.3% were missed, underscoring that static thresholds are insufficient to ensure item quality. This study provides empirical evidence supporting the integration of quantitative indices with expert review in exam evaluation. Incorporating items where the keyed answer has a p-value less than or similar to distractors may improve detection but increase workload, reflecting a trade-off between accuracy and feasibility.

Figures