Show Me What You Don't Know: Efficient Sampling from Invariant Sets for Model Validation

TL;DR

This paper introduces a training-free, diffusion-based method to sample from invariant feature sets of models, enabling visualization and analysis of model invariances without extensive training.

Contribution

It proposes a novel, training-free approach using pretrained diffusion models to efficiently sample from invariance fibers for model validation.

Findings

Reveals invariances in models like ResNet, DINO, BiomedClip

Identifies concerning invariances such as in medical imaging

Demonstrates efficiency over traditional invariance learning methods

Abstract

The performance of machine learning models is determined by the quality of their learned features. They should be invariant under irrelevant data variation but sensitive to task-relevant details. To visualize whether this is the case, we propose a method to analyze feature extractors by sampling from their fibers -- equivalence classes defined by their invariances -- given an arbitrary representative. Unlike existing work where a dedicated generative model is trained for each feature detector, our algorithm is training-free and exploits a pretrained diffusion or flow-matching model as a prior. The fiber loss -- which penalizes mismatch in features -- guides the denoising process toward the desired equivalence class, via non-linear diffusion trajectory matching. This replaces days of training for invariance learning with a single guided generation procedure at comparable fidelity. Experiments on popular datasets (ImageNet, CheXpert) and model types (ResNet, DINO, BiomedClip) demonstrate that our framework can reveal invariances ranging from very desirable to concerning behaviour. For instance, we show how Qwen-2B places patients with situs inversus (heart on the right side) in the same fiber as typical anatomy.