Absence of item origin bias on a Brazilian interinstitutional Progress Test examination: A pooled analysis of items approach

TL;DR

This study found no evidence of bias in a Brazilian interinstitutional exam based on the school where test items were created.

Contribution

The study demonstrates that best practices in test item creation can eliminate origin-based bias in cross-institutional exams.

Findings

No psychometric differences were found between self and non-self items across seven schools.

Three schools performed well on both self and non-self items, indicating no origin bias.

Confidence intervals for self and non-self items overlapped across all schools.

Abstract

It has been proposed that the school origin of items for cross-institutional Progress Tests (PTs) may introduce a bias in favour of students from the same school, posing a potential threat to the validity and reliability of PT results and cross-institutional comparisons. The aim of this study was to examine whether origin bias is present in a Brazilian cross-institutional PT examination.

This study conducted a cross-sectional analysis of seven schools affiliated with the oldest PT consortium in Brazil, utilising a pooled analysis of differences in students’ performance concerning self and non-self items. A proportional meta-analysis of the items’ rate differences and confidence intervals with random effects was performed, providing an odds ratio (OR) for self and non-self items. Differences between the two groups of items were assessed by scrutinising whether the OR and 95% confidence intervals overlapped.

The findings indicated no discernible differences in psychometric indices based on the school responsible for item creation. Three schools consistently demonstrate superior performance on items authored by their faculty, however, these they also excelled on non-self items. Furthermore, an overlap in the 95% confidence intervals for both self and non-self items was observed across all seven schools.

In contrast to prior reports, this study revealed the absence of origin bias, suggesting that adoption of best practices in blueprinting, item writing, and editing may have played a role in mitigating such bias.

Figures