A two-phase study examining perspectives and use of quantitative methods in PER

TL;DR

This study investigates the quality and reporting issues in quantitative physics education research through expert interviews and analysis of 72 manuscripts, highlighting persistent problems and recent improvements.

Contribution

It provides a systematic examination of reporting practices in PER and offers guidance for improving the clarity and validity of quantitative studies.

Findings

Limited reporting of sample and context details persists.

Reporting of limitations and cautious conclusions has improved over time.

Recommendations for better research practices are proposed.

Abstract

While other fields such as statistics and education have examined various issues with quantitative work, few studies in physics education research (PER) have done so. We conducted a two-phase study to identify and to understand the extent of these issues in quantitative PER . During Phase 1, we conducted a focus group of three experts in this area, followed by six interviews. Subsequent interviews refined our plan. Both the focus group and interviews revealed issues regarding the lack of details in sample descriptions, lack of institutional/course contextual information, lack of reporting on limitation, and overgeneralization or overstatement of conclusions. During Phase 2, we examined 72 manuscripts that used four conceptual or attitudinal assessments (Force Concept Inventory, Conceptual Survey of Electricity and Magnetism, Brief Electricity and Magnetism Assessment, and Colorado Learning Attitudes about Science Survey). Manuscripts were coded on whether they featured various sample descriptions, institutional/course context information, limitations, and whether they overgeneralized conclusions. We also analyzed the data to see if reporting has changed from the earlier periods to more recent times. We found that not much has changed regarding sample descriptions and institutional/course context information, but reporting and overgeneralizing conclusions has improved over time. We offer some questions for researchers, reviewers, and readers in PER to consider when conducting or using quantitative work.