The Effect of Transparency on Students' Perceptions of AI Graders

TL;DR

This study investigates how transparency in AI autograders influences students' perceptions, finding increased trust and discussion willingness but no change in grading willingness, possibly due to already high trust levels.

Contribution

It provides empirical evidence on the effects of transparency in AI autograders and highlights the importance of context in student trust and acceptance.

Findings

Transparency increased perceived accuracy.

Transparency boosted willingness to discuss autograders.

No effect on willingness to be graded by autograders.

Abstract

The development of effective autograders is key for scaling assessment and feedback. While NLP based autograding systems for open-ended response questions have been found to be beneficial for providing immediate feedback, autograders are not always liked, understood, or trusted by students. Our research tested the effect of transparency on students' attitudes towards autograders. Transparent autograders increased students' perceptions of autograder accuracy and willingness to discuss autograders in survey comments, but did not improve other related attitudes -- such as willingness to be graded by them on a test -- relative to the control without transparency. However, this lack of impact may be due to higher measured student trust towards autograders in this study than in prior work in the field. We briefly discuss possible reasons for this trend.