Learning Signal-Agnostic Manifolds of Neural Fields

TL;DR

This paper introduces GEM, a modality-agnostic neural manifold model that captures the underlying structure of diverse datasets, enabling interpolation, completion, and generation across different signal types.

Contribution

GEM is a novel neural manifold model that learns a low-dimensional, locally linear, and isometric representation of data across multiple modalities without requiring modality-specific architectures.

Findings

GEM enables perceptually consistent interpolation between samples.

GEM can recover diverse completions and cross-modal hallucinations.

GEM allows generation of new samples by traversing the learned manifold.

Abstract

Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives to process different classes of signals. We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner. We cast our task as one of learning a manifold, where we aim to infer a low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model -- dubbed GEM -- learns to capture the underlying structure of datasets across modalities. We can then travel along linear regions of our manifold to obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio or image signals. Finally, we show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains. Code and additional results are available at https://yilundu.github.io/gem/.