Separating Shortcut Transition from Cross-Family OOD Failure in a Minimal Model

TL;DR

This paper analyzes how shortcut features influence out-of-distribution failure in a minimal binary model, distinguishing between shortcut attraction, rule transition, and failure modes.

Contribution

It introduces a minimal model that separates shortcut correlation effects from cross-family OOD failure, providing analytic insights into these phenomena.

Findings

Ridge regularization prevents deterministic OOD failure in the deterministic regime.

Shortcut rule transition occurs when shortcut signal exceeds invariant signal in noisy settings.

Different held-out families exhibit varying error behaviors depending on shortcut correlation.

Abstract

Shortcut features are often invoked to explain out-of-distribution (OOD) failure, but training correlation, learned shortcut use, and test-time failure need not coincide. We study a minimal binary model with one invariant coordinate and one family-dependent shortcut coordinate. In the deterministic regime, positive average shortcut correlation pulls logistic ERM toward positive shortcut weight, but ridge regularization keeps the classifier invariant-dominated and prevents deterministic OOD failure. When the invariant coordinate is noisy, ridge-logistic ERM switches to the shortcut rule once the training shortcut signal exceeds the invariant signal. Whether that transition causes failure depends on the held-out family: weaker shortcut correlation yields positive excess risk, and sign-flipped families yield above-chance error. Synthetic checks match these analytic regimes and show that the same training-side transition can have different held-out consequences. The model separates shortcut attraction, shortcut-rule transition, and cross-family OOD failure.