Identifying Differences In Diagnostic Skills Between Physics Students: Developing A Rubric

TL;DR

This paper presents the development of a reliable grading rubric to assess self-diagnosis skills in introductory physics students, focusing on content knowledge and problem-solving strategies, with high inter-rater reliability.

Contribution

It introduces a new rubric for evaluating physics students' diagnostic skills, incorporating content, analysis, planning, and presentation, adaptable to various problems.

Findings

Inter-rater reliability exceeds 80%

Rubric effectively assesses diagnostic skills

Adaptable to different physics problems

Abstract

Expert problem solvers are characterized by continuous evaluation of their progress towards a solution. One characteristic of expertise is self-diagnosis directed towards elaboration of the solvers' conceptual understanding, knowledge organization or strategic approach. "Self-diagnosis tasks" aim at fostering diagnostic behavior by explicitly requiring students to present diagnosis as part of the activity of reviewing their problem solutions. We have been investigating how introductory physics students perform in such tasks. Developing a robust rubric is essential for objective evaluation of students' self-diagnosis skills. We discuss the development of a grading rubric that takes into account introductory physics students' content knowledge as well as analysis, planning and presentation skills. Using this rubric, we have found the inter-rater reliability to be better than 80%. The rubric can easily be adapted to other problems, as will be discussed in a companion paper.