The impact of incoming preparation and demographics on performance in Physics I: a multi-institution comparison

TL;DR

This study shows that incoming math preparation, rather than demographic factors, predicts student performance in Physics I across multiple institutions, emphasizing the importance of tailored instruction and comprehensive assessment of student readiness.

Contribution

It demonstrates that demographic gaps in physics exam performance disappear when accounting for math preparation, highlighting the need for multi-variable analysis in educational research.

Findings

Math SAT/ACT scores and pre-course concept inventories predict 20-30% of performance variation.

Demographic variables are not significant predictors once preparation is considered.

Performance gaps are primarily due to preparation differences, not demographic factors.

Abstract

We have studied the impact of incoming preparation and demographic variables on student performance on the final exam in physics 1, the standard introductory, calculus-based mechanics course This was done at three different institutions using multivariable regression analysis to determine the extent to which exam scores can be predicted by a variety of variables that are available to most faculty and departments. We have found that the results are surprisingly consistent across the institutions, with the only two variables that have predictive power being math SAT/ACT scores and concept inventory pre-scores. The importance of both variables is comparable and fairly similar across the institutions. They explain 20 - 30 percent of the variation in students' performance on the final exam. Most notably, the demographic variables (gender, under-represented minority, first generation to attend college) are not significant. In all cases, although there appear to be gaps in exam performance if one considers only the demographic variable, once these two proxies of incoming preparation are included in the model, there is no longer a demographic gap. There is only a preparation gap that applies equally across the entire student population. This work shows that to properly understand differences in student performance across a diverse population, and hence to design more effective instruction, it is important to do statistical analyses that take multiple variables into account. It also illustrates the importance of having measures that are sensitive to both subject specific and more general preparation. The results suggest that better matching of the course design and teaching to the incoming student preparation will likely be the most effective way to eliminate observed performance gaps across demographic groups while also improving the success of all students.