On Equivariant Model Selection through the Lens of Uncertainty

TL;DR

This paper investigates how uncertainty measures can guide the selection of equivariant models, highlighting the strengths and limitations of different uncertainty metrics in predicting model performance.

Contribution

It compares frequentist, Bayesian, and calibration-based uncertainty measures for model selection in equivariant models, revealing their effectiveness and limitations.

Findings

Uncertainty metrics generally correlate with predictive performance.

Bayesian model evidence shows inconsistent correlation.

Mismatch in complexity notions affects Bayesian measure reliability.

Abstract

Equivariant models leverage prior knowledge on symmetries to improve predictive performance, but misspecified architectural constraints can harm it instead. While work has explored learning or relaxing constraints, selecting among pretrained models with varying symmetry biases remains challenging. We examine this model selection task from an uncertainty-aware perspective, comparing frequentist (via Conformal Prediction), Bayesian (via the marginal likelihood), and calibration-based measures to naive error-based evaluation. We find that uncertainty metrics generally align with predictive performance, but Bayesian model evidence does so inconsistently. We attribute this to a mismatch in Bayesian and geometric notions of model complexity for the employed last-layer Laplace approximation, and discuss possible remedies. Our findings point towards the potential of uncertainty in guiding symmetry-aware model selection.