Refactor Analysis: Predictive Evaluations of Factor Models and Dimensionality

TL;DR

This paper introduces Refactor and Verifactor analyses, new data-driven methods for evaluating the true unidimensionality of factor models by directly assessing their ability to predict original response matrices, revealing limitations of traditional fit measures.

Contribution

The paper proposes a novel, data-first evaluation paradigm for unidimensionality that improves upon traditional correlation matrix assessments by focusing on response prediction and generalization.

Findings

Refactor metrics align with classical indices in ideal rank-1 data.

Traditional fit measures are weakly related to data recoverability in real datasets.

Quadrant correlation is a robust alternative for assessing dependence structures.

Abstract

Unidimensional factor models justify some of the most consequential summaries in science -- single scores, single ranks, and single leaderboards -- yet unidimensionality is usually assessed indirectly by fitting and evaluating models on images of the data (e.g., correlation matrices) rather than on the response matrix itself. We introduce Refactor analysis, a data-first evaluation paradigm that converts a one-factor solution into a rank-1 prediction of the original matrix by estimating both respondent- and item-side structure from dual association images. We further introduce Verifactor analysis, which evaluates the same construction under bi-cross-validated (BCV) row-column partitions for improved generalization. In simulations where the data-generating mechanism is truly rank-1 and correlational, Refactor metrics align with classical unidimensionality indices, validating the approach. However, across 200 public dichotomous datasets, traditional fit and unidimensionality measures, though highly intercorrelated, are weakly related to data recoverability, especially out of sample. This gap exposes a methodological vulnerability: excellent image-based fit can coexist with poor data-level explanatory power. Finally, treating the association measure itself as a testable hypothesis, we compare $\phi$, tetrachoric, and quadrant correlation, $q^\prime$, an important reintroduction. Quadrant correlation emerges as a simple, interpretable, and remarkably robust alternative, yielding consistently stronger reconstruction and more stable behavior under sample-size variation than commonly used correlations. Together, Refactor and Verifactor shift unidimensionality assessment from "does a one-factor model fit the correlation matrix?" to the question that matters for measurement and benchmarking: does a one-factor dependence structure recover and generalize the observed responses?