This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse

TL;DR

This paper introduces Adaptive Manifold Prototypes (AMP), a novel Riemannian optimization framework on the Stiefel manifold that enhances interpretability and accuracy in prototype networks by preventing collapse and capturing class-specific features.

Contribution

We propose AMP, a Riemannian optimization-based approach on the Stiefel manifold that improves prototype interpretability and classification performance by avoiding collapse and learning class-specific ranks.

Findings

AMP achieves state-of-the-art accuracy on fine-grained benchmarks.

AMP significantly improves causal faithfulness over prior models.

AMP effectively prevents prototype collapse through orthonormal bases.

Abstract

Prototype networks provide an intrinsic case based explanation mechanism, but their interpretability is often undermined by prototype collapse, where multiple prototypes degenerate to highly redundant evidence. We attribute this failure mode to the terminal dynamics of Neural Collapse, where cross entropy optimization suppresses intra class variance and drives class conditional features toward a low dimensional limit. To mitigate this, we propose Adaptive Manifold Prototypes (AMP), a framework that leverages Riemannian optimization on the Stiefel manifold to represent class prototypes as orthonormal bases and make rank one prototype collapse infeasible by construction. AMP further learns class specific effective rank via a proximal gradient update on a nonnegative capacity vector, and introduces spatial regularizers that reduce rotational ambiguity and encourage localized, non overlapping part evidence. Extensive experiments on fine-grained benchmarks demonstrate that AMP achieves state-of-the-art classification accuracy while significantly improving causal faithfulness over prior interpretable models.