Neural Feature Learning in Function Space

TL;DR

This paper introduces a new framework for neural feature learning in function space, unifying dependence and feature representations, and provides algorithms for optimal feature extraction with applications in multivariate learning.

Contribution

It develops a novel function-space framework for neural feature learning, connecting statistical dependence with feature representations and proposing systematic algorithms for optimal feature extraction.

Findings

Defines feature geometry unifying dependence and features in function space

Proposes nesting technique for systematic algorithm design

Demonstrates applications in conditional inference and multimodal learning

Abstract

We present a novel framework for learning system design with neural feature extractors. First, we introduce the feature geometry, which unifies statistical dependence and feature representations in a function space equipped with inner products. This connection defines function-space concepts on statistical dependence, such as norms, orthogonal projection, and spectral decomposition, exhibiting clear operational meanings. In particular, we associate each learning setting with a dependence component and formulate learning tasks as finding corresponding feature approximations. We propose a nesting technique, which provides systematic algorithm designs for learning the optimal features from data samples with off-the-shelf network architectures and optimizers. We further demonstrate multivariate learning applications, including conditional inference and multimodal learning, where we present the optimal features and reveal their connections to classical approaches.