Diagnosing Generalization Failures from Representational Geometry Markers

TL;DR

This paper introduces a top-down approach using representational geometry markers to predict and diagnose generalization failures in AI models, especially for out-of-distribution scenarios, improving model robustness and interpretability.

Contribution

It proposes a novel geometric marker-based method to forecast model performance on unseen data, moving beyond mechanistic explanations to system-level indicators.

Findings

Geometric properties of in-distribution object manifolds predict out-of-distribution performance.

Reductions in effective manifold dimensionality and utility forecast weaker OOD generalization.

Geometric patterns outperform ID accuracy in predicting transfer learning success.

Abstract

Generalization, the ability to perform well beyond the training context, is a hallmark of biological and artificial intelligence, yet anticipating unseen failures remains a central challenge. Conventional approaches often take a ``bottom-up'' mechanistic route by reverse-engineering interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-world deployment. Here, we propose using a ``top-down'' approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that serve as robust indicators of a model's future performance. Rather than mapping out detailed internal mechanisms, we systematically design and test network markers to probe structure, function links, identify prognostic indicators, and validate predictions in real-world settings. In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently forecast poor out-of-distribution (OOD) generalization. In particular, reductions in two geometric measures, effective manifold dimensionality and utility, predict weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrained models. We consistently find that the same geometric patterns predict OOD transfer performance more reliably than ID accuracy. This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.