Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis

TL;DR

The paper introduces a training-free, post-hoc CCA-based method to select and refine shared semantic features from pre-trained image encoders, significantly reducing dimensionality while improving downstream task performance.

Contribution

It presents a novel cross-model agreement technique using CCA for representation selection and dimensionality reduction, outperforming PCA in efficiency and effectiveness.

Findings

Reduces representation dimensionality by over 75% with improved performance.

Achieves accuracy gains of up to 12.6% on benchmarks.

Consistently outperforms baseline and PCA-projected representations.

Abstract

Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.