Meeting an absolute test information target with optimal number of test items via Grand Canonical Monte Carlo simulation

TL;DR

This paper introduces a Grand Canonical Monte Carlo algorithm for automated test assembly that optimally selects test forms with desired information levels and varying number of items, improving test design efficiency.

Contribution

The work presents a novel Monte Carlo method that dynamically varies test size N to find optimal test forms meeting specific information criteria.

Findings

The algorithm effectively identifies tests with minimal deviation from target information.

It determines the optimal number of items N for maximal test forms meeting the criteria.

The method adapts to different test design needs by varying N during simulation.

Abstract

This work studies IRT-based Automated Test Assembly (ATA) of multiple test forms (tests) that meet an absolute target information function, i.e. selecting from an item bank only the tests that have information functions that are at a small distance away from the target. The authors introduce the quantities multiplicity of tests and probability of selecting a test with particular number of items N and distance E from the target. A Grand Canonical Monte Carlo test-assembly algorithm is proposed that selects tests according to this probability. The algorithm allows N to vary during the simulation. This work demonstrates that the number of tests that meet the target depends strongly on N. The algorithm is capable of finding tests with small values of E and various values of N depending on the need of the test constructor. Most importantly, it can determine the optimal N for which a maximal number of tests with certain specified small E exists.