On the Proper Use of Statistical Analyses; a Comment on "Evaluation of Colorado Learning Attitudes about Science Survey" by Douglas et al

TL;DR

This paper critiques the inappropriate statistical analysis used in a study proposing a shorter version of the CLASS survey, emphasizing the importance of combining cognitive science insights with statistical methods for valid formative assessments.

Contribution

It highlights the limitations of standard factor analysis for complex constructs like the CLASS and advocates for integrating cognitive science approaches in test development.

Findings

The proposed shorter CLASS may measure a different, less useful construct.

Standard psychometric techniques may not be suitable for complex educational assessments.

Combining cognitive science with statistical analysis enhances test validity.

Abstract

The paper "Evaluation of Colorado Learning Attitudes about Science Survey" [1] proposes a new, much shorter, version of the CLASS based on standard factor analysis. In this comment we explain why we believe the analysis that is used is inappropriate, and the proposed modified CLASS will be measuring something quite different, and less useful, than the original. The CLASS was based on extensive interviews with students and is intended to be a formative measurement of instruction that is probing a much more complex construct and with different goals than what is handled with classic psychometrics. We are writing this comment to reiterate the value of combining techniques of cognitive science with statistical analyses as described in detail in Adams & Wieman, 2011 [2] when developing a test of expert-like thinking for use in formative assessment. This type of approach is also called for by the National Research Council in a recent report [3].