Neural Eigenfunctions Are Structured Representation Learners

TL;DR

This paper presents Neural Eigenmap, a neural network-based spectral method that produces structured, adaptive-length representations, improving efficiency and effectiveness in image retrieval and graph node classification tasks.

Contribution

It introduces Neural Eigenmap, a parametric spectral method that generates ordered, structured representations, enabling shorter codes with comparable performance to existing self-supervised methods.

Findings

Achieves up to 16x shorter representations in image retrieval.

Demonstrates strong results on large-scale graph node classification.

Shows that NeuralEF aligns with self-supervised learning objectives.

Abstract

This paper introduces a structured, adaptive-length deep representation called Neural Eigenmap. Unlike prior spectral methods such as Laplacian Eigenmap that operate in a nonparametric manner, Neural Eigenmap leverages NeuralEF to parametrically model eigenfunctions using a neural network. We show that, when the eigenfunction is derived from positive relations in a data augmentation setup, applying NeuralEF results in an objective function that resembles those of popular self-supervised learning methods, with an additional symmetry-breaking property that leads to \emph{structured} representations where features are ordered by importance. We demonstrate using such representations as adaptive-length codes in image retrieval systems. By truncation according to feature importance, our method requires up to $16\times$ shorter representation length than leading self-supervised learning ones to achieve similar retrieval performance. We further apply our method to graph data and report strong results on a node representation learning benchmark with more than one million nodes.