Comparing student performance on a multi-attempt asynchronous assessment to a single-attempt synchronous assessment in introductory level physics

TL;DR

This study evaluates whether multi-attempt asynchronous assessments can effectively replace traditional single-attempt synchronous exams in introductory physics, finding strong correlation for conceptual questions with five attempts.

Contribution

It introduces a method to compare multi-attempt asynchronous quizzes with single-attempt exams using isomorphic problem banks and statistical correlation measures.

Findings

Multi-attempt quizzes with five attempts correlate well with single-attempt exam performance for conceptual questions.

Numerical questions show weaker correlation beyond two quiz attempts.

Multi-attempt assessments could enhance accessibility without compromising assessment validity.

Abstract

The current paper examines the possibility of replacing conventional synchronous single-attempt exam with more flexible and accessible multi-attempt asynchronous assessments in introductory-level physics by using large isomorphic problem banks. We compared student's performance on both numeric and conceptual problems administered on a multi-attempt, asynchronous quiz to their performance on isomorphic problems administered on a subsequent single-attempt, synchronous exam. We computed the phi coefficient and the McNemar's test statistic for the correlation matrix between paired problems on both assessments as a function of the number of attempts considered on the quiz. We found that for the conceptual problems, a multi-attempt quiz with five allowed attempts could potentially replace similar problems on a single-attempt exam, while there was a much weaker association for the numerical questions beyond two quiz attempts.