Rubric Design for Separating the Roles of Open-Ended Assessments

TL;DR

This paper introduces a separable rubric for open-ended physics assessments that enables reliable scoring by untrained graders and facilitates detailed analysis of student difficulties, improving feedback and research insights.

Contribution

The paper presents a novel rubric design that separates scoring from diagnostic analysis, demonstrated through its application to the Colorado Classical Mechanics/Math Methods Instrument.

Findings

High inter-rater reliability achieved with untrained graders

Rubric effectively uncovers common student difficulties

Facilitates meaningful feedback for instruction and research

Abstract

End-of-course assessments play important roles in the ongoing attempt to improve instruction in physics courses. Comparison of students' performance on assessments before and after instruction gives a measure of student learning. In addition, analysis of students' answers to assessment items provides insight into students' difficulties with specific concepts and practices. While open-ended assessments scored with detailed rubrics provide useful information about student reasoning to researchers, end users need to score students' responses so that they may obtain meaningful feedback on their instruction. One solution that satisfies end users and researchers is a grading rubric that separates scoring student work and uncovering student difficulties. We have constructed a separable rubric for the Colorado Classical Mechanics/Math Methods Instrument that has been used by untrained graders to score the assessment reliably, and by researchers to unpack common student difficulties. Here we present rubric development, measures of inter-rater reliability, and some uncovered student difficulties.