Student behavior and test security in online conceptual assessment

TL;DR

This study investigates student behaviors and test security in online physics assessments, finding that online RBAs have comparable scores to paper versions and increased participation, with minimal impact from security concerns.

Contribution

The paper introduces a pilot study using embedded JavaScript in online RBAs to analyze student behaviors and assess test security, comparing results with traditional paper-based assessments.

Findings

Browser loss of focus was the most common online behavior.

No significant correlation between online behaviors and scores.

Participation rates increased with online administration.

Abstract

Historically, the implementation of research-based assessments (RBAs) has been a driver of education change within physics and helped motivate adoption of interactive engagement pedagogies. Until recently, RBAs were given to students exclusively on paper and in-class; however, this approach has important drawbacks including decentralized data collection and the need to sacrifice class time. Recently, some RBAs have been moved to online platforms to address these limitations. Yet, online RBAs present new concerns such as student participation rates, test security, and students' use of outside resources. Here, we report on a pilot study addressing these concerns. We gave two upper-division RBAs to courses at five institutions; the RBAs were hosted online and featured embedded JavaScript code which collected information on students' behaviors (e.g., copying text, printing). With these data, we examine the prevalence of these behaviors, and their correlation with students' scores, to determine if online and paper-based RBAs are comparable. We find that browser loss of focus is the most common online behavior while copying and printing events were rarer.We found no statistically significant correlation between any of these online behaviors and students scores. We also found that participation rates for our upper-division population went up when the RBA was given online. These results indicates that, for our upper-division population, scores on online administrations of these RBAs were comparable to in-class versions.