A tale of two test setups: The effect of randomizing a popular conceptual survey in electricity and magnetism

TL;DR

This study analyzes how different online testing setups, including randomization of options and question order, affect the results of the BEMA physics assessment, revealing that question order shuffling impacts scores but option randomization does not.

Contribution

The paper provides new insights into the effects of test randomization methods on assessment outcomes and introduces methods to distinguish genuine responses from guesses.

Findings

Question order shuffling affects BEMA scores.

Option randomization alone does not significantly impact scores.

Data cleaning based on time-in-testing is essential for valid analysis.

Abstract

We describe a retrospective study of the responses to the Brief Electricity and Magnetism Assessment (BEMA) collected from a large population of 3480 students at a large public university. Two different online testing setups were employed for administering the BEMA. Our tale focuses on the interpretation of the data collected from these two testing setups. Our study brings new insight and awareness to the effect of the randomization on the BEMA test results. Starting from an a priori common sense model, we show simple methods to detect and separate guessing from the genuinely thought responses. In addition, we show that the group of responses with low scores (7 or less out of 30) had different answer choice strategies than the groups with average or high scores, and that the time-in-testing is an essential parameter to be reported. Our results suggest that the data should be cleaned to insure that only valid times-in-testing are included before reporting and comparing statistics. We analyze in detail the effect of the answer-options randomization and the effect of the shuffling of the questions order. Our study did not detect any significant effect of the option randomization alone on the BEMA scores, for valid times-in-testing. We found clear evidence that shuffling the order of the independent questions does not affect the scores of the BEMA test, while shuffling the order of the dependent questions does.