Eigenfunction Extraction for Ordered Representation Learning

TL;DR

This paper introduces a framework for extracting ordered, identifiable eigenfunctions from kernels in representation learning, enabling better feature importance understanding and efficient adaptive representations.

Contribution

It proposes a modular framework for eigenfunction extraction compatible with modern kernels, unifying low-rank approximation and Rayleigh quotient methods.

Findings

Eigenvalues serve as effective feature importance scores.

The approach improves feature selection and efficiency in real-world datasets.

Validated on synthetic and image datasets with promising results.

Abstract

Recent advances in representation learning reveal that widely used objectives, such as contrastive and non-contrastive, implicitly perform spectral decomposition of a contextual kernel, induced by the relationship between inputs and their contexts. Yet, these methods recover only the linear span of top eigenfunctions of the kernel, whereas exact spectral decomposition is essential for understanding feature ordering and importance. In this work, we propose a general framework to extract ordered and identifiable eigenfunctions, based on modular building blocks designed to satisfy key desiderata, including compatibility with the contextual kernel and scalability to modern settings. We then show how two main methodological paradigms, low-rank approximation and Rayleigh quotient optimization, align with this framework for eigenfunction extraction. Finally, we validate our approach on synthetic kernels and demonstrate on real-world image datasets that the recovered eigenvalues act as effective importance scores for feature selection, enabling principled efficiency-accuracy tradeoffs via adaptive-dimensional representations.