Analytics of Adaptive Online Testing in Practice Over a Decade

TL;DR

This paper analyzes the practical performance and characteristics of adaptive online testing systems over a decade, focusing on how they adjust to examinee ability and the implications of test length on their effectiveness.

Contribution

It provides empirical insights into the behavior of adaptive testing in real-world settings, highlighting how test length influences the adaptive mechanism's observable effects.

Findings

Adaptive tests can effectively reflect ability even with few items.

Probability of correct answers approaches 0.5 with large test lengths.

Short tests still demonstrate adaptive behavior through item difficulty and ability relationships.

Abstract

Adaptive online testing efficiently assesses examinee proficiency by dynamically adjusting the difficulty of test items based on their performance. To achieve this, items are selected so that their difficulty closely matches the test taker's estimated ability at each stage of the test. This alignment implies that the probability of a correct answer tends toward 0.5. However, in practical settings, this probability may not converge to 0.5 unless the test comprises a sufficiently large number of items. This could give the impression that the adaptive mechanism is not functioning properly. Nevertheless, even when the number of items is small, such as 5 or 7, the adaptive nature of the system can still be observed by examining the relationship between item difficulty and the mean estimated ability of examinees for the corresponding item. Considering that each item typically requires about 2-5 minutes to solve, this number of items appears to be a practical choice that avoids excessive fatigue for test-takers.